Self-Monitoring Cluster of Network Security Devices

ABSTRACT

A computing device may be joined to a cluster by discovering the device, determining whether the device is eligible to join the cluster, configuring the device, and assigning the device a cluster role. A device may be assigned to act as a cluster master, backup master, active device, standby device, or another role. The cluster master may be configured to assign tasks, such as network flow processing to the cluster devices. The cluster master and backup master may maintain global, run-time synchronization data pertaining to each of the network flows, shared resources, cluster configuration, and the like. The devices within the cluster may monitor one another. Monitoring may include transmitting status messages comprising indicators of device health to the other devices in the cluster. In the event a device satisfies failover conditions, a failover operation to replace the device with another standby device, may be performed.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Application No.61/139,078, filed Dec. 19, 2008, and entitled, “Clustered Architecturefor Network Security Devices,” which is hereby incorporated by referencein its entirety.

TECHNICAL FIELD

This disclosure relates to network services and, in particular, toformation of a cluster comprising two or more computing devicesconfigured to provide network services.

BRIEF DESCRIPTION OF THE DRAWINGS

Additional aspects and advantages will be apparent from the followingdetailed description of preferred embodiments, which proceeds withreference to the accompanying drawings.

FIG. 1 is a block diagram of one embodiment of a cluster;

FIG. 2A is a state diagram depicting a method for adding a device to acluster;

FIG. 2B is a flow diagram of one embodiment of a method for adding adevice to a cluster;

FIG. 3A depicts the relationships and/or transitions between clusterdevice operational modes;

FIG. 3B depicts data flow between cluster devices;

FIG. 3C is a flow diagram of one embodiment of a method for monitoringcluster devices;

FIG. 3D is a flow diagram of one embodiment of a method for performing acluster failover operation;

FIG. 4 is a block diagram of one embodiment of a cluster device;

FIG. 5 is a flow diagram depicting one embodiment of a method forassigning a flow to a cluster device;

FIG. 6A is a block diagram depicting one example of related flowassignment, in which related forward and reverse flows are assigned tothe same cluster device;

FIG. 6B is a block diagram depicting another example of related flowassignment, in which related flows are assigned to the same clusterdevice;

FIG. 6C is a block diagram depicting an example of security flowassignment, in which flows associated with the same tunnel are assignedto the same cluster device;

FIG. 6D is a block diagram depicting another example of security flowassignment, in which flows associated with the same inbound or outboundsecurity association are assigned to the same cluster device, whereasthe inbound and/or outbound tunnel flows may be assigned to differentcluster devices;

FIG. 6E is a block diagram depicting another example of security flowassignment, in which flows related to the same security association(inbound and outbound) are assigned to the same device;

FIG. 6F is a block diagram depicting another example of related flowassignment, in which the flows related to a tunnel switch are assignedto the same device;

FIG. 7 is a block diagram illustrating one embodiment of a clustercomprising a shared Internet Key Exchange module;

FIG. 8 is a block diagram illustrating one embodiment of a clustercomprising distributed Internet Key Exchange modules;

FIG. 9A is a block diagram depicting an example of flow assignment in acluster comprising a shared Internet Key Exchange module;

FIG. 9B is a block diagram depicting an example of flow assignment in acluster comprising distributed Internet Key Exchange modules;

FIG. 10 is a flow diagram of another embodiment of a method forassigning network flows to cluster devices.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

As used herein, a clustered computing system or “cluster” may refer totwo or more computing devices configured to cooperatively perform atask. In some embodiments, a cluster may be formed of a plurality ofcomputing devices of the same type (e.g., a homogeneous cluster).Alternatively, a cluster may comprise computing devices of differenttypes and/or configurations (e.g., a heterogeneous cluster).

A cluster may be configured to provide network communications andsecurity services including, but not limited to: providing firewallservices, acting as a forward and/or reverse proxy, virtual privatenetworking (VPN), packet filtering, anti-virus services, InternetProvider Security (IPS), tunneling, Spam blocking, Web blocking, and thelike.

A cluster may be configured to operate in a “load balancing” or “highthroughput” mode. As used herein, “load sharing” or “high throughput”may refer to an operational mode of a cluster in which one or more ofthe computing devices in the cluster are configured to work together toimplement one or more tasks or services. For example, a highly complexcomputational task may be split up into a plurality of different parts,each of which may be performed by a different computing device in thecluster. Alternatively, or in addition, a set of tasks may bedistributed to a plurality of different computing devices in the cluster(e.g., each of a plurality of different VPN connections may be servicedby different members of the cluster). Since the load represented by thetask(s) or service(s) may be shared among the computing devices in thecluster, the cluster may be capable of performing certain task(s) and/orproviding certain service(s) more efficiently than a single computingdevice.

One or more of the computing devices in a cluster may be configured tooperate in “high availability mode.” In high availability mode, one ormore of the computing devices in the cluster may be in “active” or“primary” mode, while other computing devices in the cluster are in“standby” or “secondary” mode. The devices in active mode may beconfigured to perform the task(s) and/or provide the service(s)implemented by the cluster. The devices may have a “static” and“working” role. The static role may be the role of the device as definedin the device-specific cluster configuration thereof, defined by alicense (or lack thereof) of the device, defined by a clusterconfiguration, or the like. The working role may be the currentoperating state or role of the device as necessitated by the operatingconditions of the cluster. A device may have a static role of “active”or “primary,” meaning that the device has its own license and mayactively process and pass network traffic. A member having a static roleof “standby” or “secondary” may not have a license and may not activelyprocess and/or pass network traffic, but operate as a backup to theother cluster device (e.g., when an active device fails over, a standbydevice may be activated to take its place).

The working role of the cluster devices may include devices that are“active,” and “standby.” Active devices include the primary devices(that are currently running), and any secondary devices that have beenactivated to process and/or pass network traffic in place of failed overprimary members. Accordingly, a cluster device operating in standby modemay not perform the task(s) and/or provide the service(s) implemented bythe cluster. A cluster device operating in standby mode may become“activated” responsive to a failure of one or more of the activecomputing devices. When activated, the device may implement the task(s)and/or service(s) that were formerly provided by the failed over device,which may prevent an interruption of cluster services (e.g., allow thecluster to provide “high availability” services).

In some embodiments, one of the computing devices in a cluster mayoperate as a “cluster master.” The cluster master may manage theconfiguration of the cluster, assign processing tasks to the clustermembers (e.g., assign network flows to cluster devices), maintaincluster state information (e.g., flow assignment table, securityinformation, etc.), manage shared resources (e.g., outbound trafficaddresses, Destination Network Address Translation (DNAT) tables, etc.),synchronize global, run-time synchronization data with a backup master,manage device failover, and the like.

Another one of the computing devices in the cluster may operate as a“backup master,” which may be configured to backup the data used by thecluster master. Accordingly, when the cluster master fails over (due toa device failure, upgrade, maintenance, or the like), the backup mastermay replace the cluster master with minimal service disruption (e.g.,may transition into the role of cluster master). When the backup masteris promoted to cluster master, another one of the cluster members may beselected as a new backup master. The cluster master may be configured tosynchronize global, run-time synchronization data with the backupmaster. The global, run-time synchronization data may include the dataneeded by the backup master to begin operating as the cluster master(e.g., cluster configuration, cluster state, and the like).

FIG. 1 is a block diagram of one example of a system comprising acluster of communicatively coupled computing devices. The cluster 110 ofFIG. 1 may be configured to provide the network communications andsecurity services described above (e.g., firewall, packet filtering,VPN, and so on).

In the FIG. 1 example, the cluster 110 comprises three computing devices112, 114, and 116. However, the disclosure is not limited in thisregard, and the cluster 110 could be configured to include any number ofcomputing devices.

The cluster 110 is configured to communicatively couple an organization130 to a network 140. The network 140 may comprise a set ofinterconnected networks that implement one or more standardcommunications protocols (e.g., the Internet Protocol Suite, TCP/IP, orthe like). Accordingly, the network 130 may comprise a collection ofnetwork infrastructures including, but not limited to: Ethernetnetworks, wireless networks, Public Switched Telephone networks (PSTN),Home networks, Wi-Fi networks, and the like.

The cluster 110 may include a network interface 120 to communicativelycouple the cluster 110 (and the organization 130) with the network 140.The cluster interface 120 may be shared among the computing devices 112,114, and 116, in the cluster 110. Accordingly, the cluster 110 mayappear as a single device to the network 130. In some embodiments, anadditional communications interface 122 (organization interface 122) maybe provided for communication with the organization 130. The networkinterface 120 and organization interface 122 may be implemented usingrespective traffic switches (or different ports of the same trafficswitch), or using other network devices (e.g., hubs, routers, switches,or the like). Accordingly, network traffic between the organization 130and the network 140 may pass through the cluster 110, which may,therefore, provide network security services to the organization,including, but not limited to: firewall, packet filtering, Spamfiltering, Web filtering, and so on. In addition, the cluster 110 mayprovide for secure access to services within the organization 130 byentities within the network 140 (e.g., provide VPN services). Forexample, a VPN may allow an entity 142 communicatively coupled to thenetwork 140 using a computing device 143 (e.g., personal computer,laptop computer, notebook computer, or the like), to access a mailserver 134 and/or file server 136 within the organization 130.

The computing devices 112, 114, and 116 in the cluster 110 may also becommunicatively coupled to one another. In some embodiments, the devices112, 114, and 116 may each be communicatively coupled using respective,dedicated cluster interface ports 111A, 111B, and 111C. The clusterinterface ports 111A, 111B, and 111C may be concentrated in clusternetwork interface 124, which may be comprise one or more routers,switches, concentrators, hubs, or the like.

The devices 112, 114, and 116 may communicate cluster-specificinformation (discussed below) using the cluster interface port 111A,111B, and 111C (via the cluster interface 124). In some embodiments, thedevices 112, 114, and 116 may be configured to implement acluster-specific protocol on the cluster interface ports 111A, 111 B,and 111C. The cluster-specific protocol may provide for fast andreliable communication of cluster-specific data, including, but notlimited to: cluster state information, service information, devicehealth information, failover, synchronization data, and the like. Insome embodiments, the cluster-specific protocol may be implementedwithin the data-link layer (of the eight layer Open SystemInterconnection Reference Model (“OSI model”)), as opposed to theapplication layer (or another, higher layer), to allow the clusterinterface port to continue to operate regardless of faults in theapplication layer of the device 112, 114, and/or 116.

The cluster 110 may be formed by designating one of the computingdevices 112, 114, or 116 to act as a cluster master. In the FIG. 1example, computing device 112 may be selected as the cluster master(e.g., by personnel or the organization, IT staff, or the like).Designing a cluster master may comprise providing the computing device112 with a cluster configuration, which may define the tasks and/orservices to be provided by the cluster 110. For example, a clusterconfiguration may determine the security services to be provided by thecluster 110 (e.g., packet filtering, VPN, etc.), define the securitypolicy to be implemented by the cluster 110, and so on. Configuring thecluster master may further comprise setting a “cluster enabled” flag onthe designated computing device 112, setting a cluster identifier (e.g.,cluster name), designating one or more ports for cluster communication(e.g., cluster interface port 111A), and so on. When the computingdevice 112 is so configured, the cluster 110 may be created (a cluster110 comprising a single computing device 111), and the computing device112 may begin acting as a cluster master.

Additional computing devices (e.g., devices 114 and 116) may be added tothe cluster 110 by connecting the device(s) to the cluster interface124, the network interface 120 and/or the organization interface 122.The cluster master 112 (and/or other devices added to the cluster), maybe configured to detect the connection of a new computing device to thecluster 110 (e.g., to the cluster interface 124, network interface 120,and/or the organization interface 122). In some embodiments, thecomputing devices in the cluster 110 (e.g., the cluster master computingdevice 112) may transmit periodic discovery messages via theirrespective cluster interface ports (e.g., cluster interface port 111A ofthe computing device 112). The discovery messages may comprisebroadcast-type messages configured to be received by any computingdevice (114 and/or 116) communicatively coupled to the cluster interface124 (or other interface 120 and/or 122).

Once discovered, the new computing device 114 may receive adevice-specific configuration from one of the other devices in thecluster 110. The device-specific configuration may configure the newcomputing device 114 to operate in “cluster” mode, configure the clusterinterface port of the device (port 111B, discussed below), assign a roleto the device (e.g., active, standby, or the like), and so on.

After receiving the device-specific cluster configuration, the newdevice 114 may join the cluster (e.g., begin communicating via thecluster interface port 111B), and receive a cluster configuration fromanother cluster device. The cluster configuration may include adefinition of the services provided by the cluster 110 (e.g., VPN,firewall, packet filtering, etc.), define a security policy implementedby the cluster 110, define cluster capabilities (e.g., maximum number ofsimultaneous connections, etc.), and the like. The new device 114 mayreceive and implement the device-specific configuration (and clusterconfiguration).

The cluster master (or other cluster device 112, 114, and/or 116) mayverify that the device has successfully implemented the device-specificconfiguration (including the cluster configuration). After theverification, the new computing device may be joined to the cluster.Joining the cluster 110 may comprise establishing and/or joining asecure cluster communications channel, which may comprise providing thenew device with a shared key, performing a key exchange protocol, or thelike. The secure connection may be established on the cluster interfaceport of the device (e.g., port 111A, 111B, or 111C). The secure clusterconnection may be used to synchronize cluster configuration data, flow,run-time synchronization data, global, run-time synchronization data,security services data, time, device monitoring information (e.g.,device status messages, health scores, etc), provide access to sharedresources (e.g., address pools, port pools, and so on), provide accessto shared services (e.g., a shared Internet Key Exchange (IKE) module),and the like.

As discussed above, a cluster 110 may include “active” members operatingin “high throughput” mode as well as “standby” member operating in “highavailability mode.” A cluster configuration may specify that the cluster110 is to include a particular number of active cluster members (e.g., Nactive members), with any additional members to operate in standby mode.In another embodiment, a cluster configuration may specify N activemembers and Y standby cluster members, specify a certain proportion ofactive to standby cluster members, or the like. As new members are addedto the cluster (according to the discovery processes above), the clustermaster (or another cluster device) may determine whether the clustershould be configured to operate in active or standby mode.

If the device is to operate in active mode, it may be made available toperform task(s) and/or provide service(s) as directed by the clustermaster (according to a load balancing scheme defined by the clusterconfiguration). If the device is to operate in standby mode, it may nottake on active task(s) and/or provide service(s) until another devicefails or stops responding.

FIG. 2A is a state diagram 200 depicting the addition of a new device toa cluster. The state diagram 200 may be implemented as a method,comprising a plurality of steps (e.g., method 201 discussed below inconjunction with FIG. 2B).

When in state 210, a device may be communicatively connected to anetwork (e.g., connected to the interfaces 120, 122, and/or 124 of FIG.1). The new device may be unconfigured (in a default or safeconfiguration) and, as such, may not yet have joined the cluster (e.g.,not configured to communicate with other cluster devices, receiveprocessing tasks, etc.). In state 210, the device may be discoverable byother devices in the cluster. Causing a device to enter state 210 mayinclude physically connecting a communications port of the device to anetwork device (e.g., switch, router, concentrator, or the like),enabling one or more network switch ports or interfaces (e.g.,interfaces of network devices 120, 122, and/or 124), enabling one ormore communications interfaces of the device, modifying a networkconfiguration and/or topology to communicatively couple the device toother cluster devices, or the like.

In state 210, the device may be actively or passively discovered byanother device. In some embodiments, other cluster devices may beconfigured to transmit discovery messages within a cluster network(e.g., the cluster master may periodically transmit broadcast messagesto a cluster interface, such as the interface 124 of FIG. 1). Thediscovery messages may include a request for the device to providedevice identifying information, such as a device serial number, version,capabilities, licensing information, and the like.

The cluster device(s) may use the information to determine whether thedevice is eligible to join the cluster (e.g., is compatible with theother devices in the cluster and/or licensed to operate in a clusteredenvironment). If the device is not compatible with the cluster and/ornot licensed for clustered operation, the device may transition to anon-member state 280. In the non-member state 280, the device may notparticipate as a primary (active) and/or secondary (standby) member ofthe cluster. In addition, the other cluster devices may be configured toexclude the device from secure cluster communications, assignment ofcluster processing tasks, and the like.

If the device is compatible with other cluster devices and/or isotherwise eligible to join the cluster, a cluster “join” procedure maybe initiated. The join procedure may transition the device into a joinstate 220. Transitioning to the join state 220 may comprise the clusterdevice(s) (e.g., the cluster master or other device) transmitting adevice specific cluster configuration to the device. The device-specificconfiguration may be loaded and implemented by the device. Loading andimplementing the device-specific configuration causes the device to“join” the cluster and transition to state 220.

When in state 220, the device may be prepared to join the cluster as anactive or standby cluster member. The preparation in state 220 mayinclude a cluster device (e.g., cluster master) validating thedevice-specific configuration, verifying a license of the device,synchronizing cluster configuration and/or run-time data with thedevice, and the like.

In state 220, if the cluster configuration is not validated (e.g., ifthe cluster configuration loaded on the device differs from the clusterconfiguration implemented by the cluster master), the device may bereturned to state 210, where the device may be re-discovered and havenew device-specific configuration data transmitted thereto (e.g., by thecluster master).

The synchronization performed in state 220 may include transmittingrun-time, global cluster configuration data to the device (e.g., fromother cluster device and/or the cluster master). The run-time, globalcluster configuration data may include data used by the devices in thecluster to process and/or pass network traffic and may include, but isnot limited to: flow table information (discussed below), securityinformation (e.g., phase one security association (P1SA), phase twosecurity association (P2SA), session keys, etc.), assigned IP for mobileVPN (MOVPN), user session information, a list of devices in the clusters(along with the static and/or working roles thereof), cluster devicestatus information (e.g., device health, etc.), cluster electioninformation (discussed below), and the like.

The cluster configuration data (synchronized from the cluster master)may be used by the cluster device to process network traffic and/orservice network requests when the device is operating in active mode. Insome embodiments, synchronization may include establishing a securecluster communications channel on a particular communication interface(e.g., on a dedicated cluster interface port, such as ports 111A-111Cdepicted in FIG. 1). As will be discussed below, the clustersynchronization channel may be configured to provide forhigh-performance data synchronization that is resistant toapplication-layer failures (e.g., implemented at a low layer of the OSImodel). If the synchronization cannot be performed (e.g., the devicefails to receive the cluster configuration data, the synchronizationchannel cannot be established, or the like), the device may transitionto the non-member state 280.

The license verification performed in state 220 may include determiningwhether licensing information of the device is valid (e.g., using acryptographic technique, such as verifying a digital signature, hashvalue, or the like). If the device does not have a license and/or haslicensing information that cannot be validated, the device may beconfigured to operate in “standby” mode (e.g., have a static role of“secondary” or standby). When in standby mode, the device may notactively perform cluster processing tasks (e.g., handle network flows).If the device has a license and/or the licensed capabilities of thecluster allow for additional active members, the device may be eligibleto be an active member of the cluster (e.g., have a static role of“primary” or active).

The license verification in state 220 may further include determiningthe licensed capabilities of the cluster. In some embodiments, thelicensed capabilities of the cluster may be determined by combining thecluster device licenses. The combination may be made in a number ofdifferent ways. In some embodiments, the licenses may be combined in a“least common capabilities” fashion, in which the features supported bythe cluster may be determined by the “minimum” set of features providedin each of the device licenses. For example, if the license of a firstprimary member of the cluster provides for features A, B, and C and thelicense of a second primary member provides for only features A and B,the cluster comprising the first and second members may only supportfeatures A and B. In some embodiments, the licenses may be combined inan “OR”-type operation (or other logical combination) in which thecapabilities of the devices are added together (e.g., a clustercomprising a first device licensed to provide features A and B and asecond device licensed to provide features B and C may be capable ofproviding features A, B, and C). Alternatively, or in addition, certainlicensing features (e.g., VPN) may require that each device has anenabling license, while others may not (e.g., operate in an “OR”fashion).

At step 220, a static role of the device may be determined. As discussedabove, if the device does not have its own license, the device may beassigned to operate in a secondary or standby role, meaning that thedevice may not act as an active cluster device (e.g., a device capableof accepting tasks from the cluster master), until failover occurs. Inaddition, if the device does have a license, but that license does notgive the device the same capabilities implemented by the cluster and/orthe cluster configuration already has its allotted number of primarymembers (e.g., as determined by the cluster configuration), the devicebe given a secondary or standby static role.

If the device has sufficient licensing privileges and/or other clusterdevices have failed over (been removed from the cluster due to a devicefailure or the like), the device may be configured to operate in aprimary or active role (e.g., as an active part of the cluster).

After verifying the device license, synchronizing cluster configurationdata, and the like, the device may transition to state 230, where it mayact as a cluster member. If the device is configured as a primary oractive cluster device, the device may begin accepting processing tasksfrom the cluster master (e.g., handling network flows, etc.). If thedevice is configured as a secondary or standby device, the device maywait until failover operation occurs before it begins accepting tasksfrom the cluster master.

The device may leave the cluster member state 230 by being deactivated(e.g., by the cluster master, a human operator, or the like), beingdeactivated for an upgrade operation, by being reset, by being failedover, or the like. If the device is deactivated, it may enter thenon-member state 280. If the device is reset, it may enter thediscoverable state 210, at which point the device may rejoin the clusteras described above.

When the device is in the non-member state 280 (due to a failure to jointhe cluster from state 220, being deactivated from the active memberstate 230, or the like), the device may not operate as a primary orsecondary cluster device. Accordingly, the device may not activelycommunicate with other cluster devices, may not accept tasks from thecluster master, and/or enter an active state if/when other clusterdevice(s) are failed over.

When in the active member state 230, the device may be configured tooperate in one of a plurality of operational roles including, but notlimited to: cluster master, backup master, and active. The clustermaster may be configured to manage the cluster, which may include, butis not limited to: maintaining a list of the devices in the cluster(e.g., a list of active and standby devices), assigning processing tasksto the active devices, maintaining global, run-time synchronizationdata, managing shared resources, assigning tasks to active clustermembers (e.g., perform a load balancing function), handling networkflows, monitoring cluster health, managing device failover (e.g.,providing for activation of standby cluster members in response to afailure of one or more of the active devices), and the like. The global,run-time synchronization data may include a flow assignment datastructure comprising a mapping between the network flows handled by thecluster and the cluster device assigned thereto, flow, run-timesynchronization data for each of the flows (e.g., session information,such as cache data, session keys, security data, and the like), sharedresource data (e.g., address pools, port pools, hostout data, and thelike), and so on.

When operating as a backup master device, the device may be configuredto receive global, run-time synchronization data from the cluster master(e.g., maintain the same set of data as the cluster master).Accordingly, the backup master may quickly take over the role of thecluster master if needed (e.g., if the cluster master fails, has itshealth score fall below a threshold, or the like).

An active cluster device may be configured to handle network flowsassigned thereto by the cluster master. In addition, an active clusterdevice may monitor the health of other cluster members (e.g., thecluster master). The cluster master and/or backup master may beconfigured to operate as active cluster devices (e.g., may processnetwork flows, monitor other cluster devices, and the like). The activecluster devices may be configured to transmit flow, run-timesynchronization data to the cluster master. The flow, run-timesynchronization data may include data pertaining to each of the networkflow(s) handled by the device. The flow, run-time synchronization datamay be used in a failover operation to allow another device to handlethe flow with minimal service disruption.

The cluster formation process described above may include selection of acluster master. In some embodiments, the cluster master may be the firstdevice added to the cluster (e.g., the first device configured tooperate in clustered mode). Alternatively, or in addition, a clustermaster may be periodically selected by a human operator (e.g., via aconfiguration interface) and/or by the devices in the cluster (e.g., inresponse to the cluster master failing, the cluster master's healthscore (discussed below) falling below a threshold, after a predeterminedtime threshold, or the like).

FIG. 2B is a flow diagram of a method 201 for adding a device (networksecurity device) to a cluster. The method 201 may be implemented on acomputing device comprising a processor and memory using one or morecomputer-readable and/or computer-executable instructions. Theinstructions comprising the method 201 may be embodied as one or moredistinct software modules, which may be stored on a computer-readablestorage medium, such as a hard disc, optical storage media, memory, orthe like. In some embodiments, one or more steps of the method 201 maybe tied to particular machine components, such as computer-readablestorage media, communications interfaces, processing modules, or thelike.

At step 211, the method 201 may be initialized, which may compriseloading one or more computer-readable instructions from one or morecomputer-readable storage media, accessing and/or initializing one ormore communications interfaces, and the like.

At step 221, the method 201 may discover a device. Discovering thedevice may comprise detecting a communications interface of the newdevice. For example, the new device be communicatively coupled tonetwork interface used by the cluster (e.g., network interface 120, 122,and/or 124 of FIG. 1), one or more communications interfaces of thedevice may be activated, a configuration of the device may be set suchthat the device is capable of communication with other cluster devicesor the like. Discovering the device at step 221 may comprise activediscovery and/or passive discovery. Active discovery may comprise themethod 201 transmitting network traffic (e.g., broadcast packets, or thelike), which may be received (and responded to) by the device. Thediscovery messages may be transmitted automatically and/or periodically.Alternatively, the method 201 may transmit discovery messages only ifinstructed to do so (e.g., via a configuration interface, an SNMPmessage, or the like). Passive discovery may comprise the method 201monitoring network traffic (e.g., for ARP requests, DHCP requests, orthe like), accessing router ARP tables, or the like to discover thedevice without actively transmitting network traffic thereto.

At step 231, the eligibility of the device to join the cluster may bedetermined. In some embodiments, determining the eligibility of thedevice to join the cluster may be based upon device-identifyinginformation, such as an indicator of the version or revision of thedevice (e.g., software version, firmware version, hardware revision,etc.), the capabilities of the device (e.g., hardware capabilities, suchas processor speed, memory, and the like, software installed, etc.),device licensing information, and so on. For example, the cluster may beconfigured to only accept certain devices (or device versions) that havecertain processing capabilities (e.g., processing speed, memorycapacity, communications interface capabilities, such as a gigabitEthernet interface, or the like). Step 231 may comprise the method 201interrogating the device to determine certain device properties (e.g.,hardware configuration, software version, firmware version, etc). If thedevice is not eligible to join the cluster (does not meet the softwareor hardware requirements for cluster membership), the flow may continueat step 281; otherwise, the method 201 may continue to step 236.

At step 236, a device-specific configuration may be transmitted to thedevice, and device implementation thereof may be validated. Thetransmission of the device-specific configuration at step 236 maycomprise selecting device-specific configuration data from a pluralityof different device-specific configurations, each of which may beadapted to particular device hardware and/or software configuration orversion. The selection may be based upon the device-identifyinginformation obtained at step 231. Step 236 may further compriseverifying that the device has implemented the device-specificconfiguration. Verification may comprise the device transmitting aconfirmation message to the method 201, the method 201 interrogating thedevice (e.g., for a hash value or other indicator of the device-specificconfiguration), or the like. If the device-specific configuration isverified at step 236, the flow may continue to step 241. Otherwise, theflow may return to step 236 where the eligibility of the device to jointhe cluster may be re-determined and/or the device-specificconfiguration may be re-transmitted. Alternatively, or after apredetermined number of device-specific configuration verificationfailures, the flow may continue to step 281.

At step 241, a cluster configuration may verified. In some embodiments,the cluster configuration may be included with the device-specificconfiguration. Alternatively, the cluster configuration may betransmitted separately (e.g., transmitted at step 241). As discussedabove, the cluster configuration may include a security policyimplemented by the cluster, identifiers of the devices in the cluster,cluster communication configuration (e.g., cluster port assignment(s),interface port assignment(s), and the like), and the like.

Step 241 may further comprise verifying that the device has implementedthe cluster configuration. The verification of step 241 may comprise thedevice transmitting a confirmation message to the method 201, the method201 actively interrogating the device, or the like. If the clusterconfiguration is verified, the flow may continue to step 251; otherwise,the flow may return to step 241, where the cluster configuration may bere-transmitted to the device and re-verified by the method 201.Alternatively, or after a threshold number of cluster configurationverification failures, the flow may continue to step 281.

At step 251, a static role of the device may be determined. Determininga static role of the device may comprise accessing a license of thedevice, evaluating the device-identifying information about the device,and so on. Accordingly, assignment of the device role in the cluster maybe determined and transmitted with the device-specific configuration atstep 236. Alternatively, the role assignment may be made in a separatestep 251 as depicted in FIG. 2B.

At step 251, if the device is not licensed, or a license of the clusterdefines a maximum number of active devices, which has already been met,the device may be assigned a static role of “secondary” or “standby.”When in the secondary or standby role, the device may not be assignedcluster processing tasks (e.g., handle network flows). If the device islicensed and/or a maximum number of active devices defined in a clusterlicense has not been met, the device may be assigned a static role of“primary” or “active.” When in the primary or active role, the devicemay be available to perform cluster processing tasks (e.g., handlenetwork flows). Assigning a role to the device may further compriseelecting the device to act as a cluster master or backup master asdescribed above. For example, if the device is the first device in thecluster, the device may be automatically selected as the cluster master.Similarly, if the cluster does not yet have a backup master, the devicemay be given the role of backup master.

In some embodiments, step 251 may further comprise determining thelicensed capabilities of the cluster. If the device has its own license,the license may be transmitted to the device implementing the method201. The license may be combined with the licenses of the other devicesin the cluster (if any). The licensed capabilities of the cluster maydefine the capabilities thereof, which may include, but are not limitedto: the number of active connections supported by the cluster, thethroughput of the cluster, the services provided by the cluster (e.g.,VPN, SSL, etc.), the number of active devices in the cluster, and so on.In some embodiments, the licenses may be combined by determining theleast common capabilities therebetween (e.g., if a first license allows500 concurrent connections, and a second license allows 700 concurrentconnections, the cluster may be licensed to the lower number ofconcurrent connections, or 500 concurrent connections). Alternatively,the combination may be additive or according to the maximum capabilitiesof the licenses. Different licensed features may be combined indifferent ways (e.g., certain capabilities may be determined accordingto least common capability, while others may be additive, and so on).

At step 261, the device may join the cluster in its assigned role (thestatic role determined at step 251). If the device has been assigned anactive role within the cluster, joining the cluster at step 261 maycomprise configuring the other members of the cluster to communicatewith the device (e.g., using a secure, cluster communications protocol),configuring the cluster master to assign processing tasks to the device(e.g., assign network flows to the device), and so on. Accordingly,joining the cluster may comprise the cluster master (or other clusterdevice) provide a shared key to the device to allow the device tosecurely communicate with other cluster devices. Alternatively, or inaddition, joining may comprise performing a key exchange operation withone or more cluster devices to establish shared keys therewith.

If the device has been selected to operate as the cluster master,joining the cluster at step 261 may comprise initializing cluster masterdata structure, such as a flow assignment data structure, global,run-time synchronization data structure, shared resource data structure,and the like. The device may be configured to receive and assign networkflows to cluster devices as described herein. In addition, the devicemay be configured to synchronize global, run-time synchronization datawith a backup master device (if any). If the device is configured tooperate as the backup master of the cluster, joining the cluster at step261 may further comprise configuring the cluster master to synchronizeglobal, run-time synchronization data with the device, which mayinclude, but is not limited to: a flow assignment data structure, flow,run-time synchronization data (data associated with each of the assignedflows), shared resource data, and the like.

If the device has been assigned a standby role within the cluster,joining the cluster at step 261 may comprise operating in standby mode(e.g., passively synchronizing with the cluster master) until devicefailover occurs, at which point the device may transition to an activerole as described above. Accordingly, joining the cluster as a standbydevice may comprise establishing a secure communications channel withthe device, configuring the other cluster devices to use the device as afailover candidate (e.g., make the device available in the event of afailure of one of the other cluster devices), synchronizing clusterconfiguration and run-time data with the device, and the like.

At step 281, if the device is ineligible or unable to join the cluster,the device may be set as a non-member. Setting a device as a non-membermay comprise configuring the device to operate in its default or “safe”configuration. In addition, other cluster devices may be configured toexclude the device from secure cluster communications, from eligibilityfor assignment of cluster processing tasks, from eligibility for use asa failover device, and the like. Accordingly, the device may notimplement the cluster configuration, communicate with other clusterdevices (e.g., have access to the secure, cluster communicationschannel), and so on. When reverted to the default or safe configuration,the device may be discoverable by other cluster members and, as such,may attempt to join the cluster at a later time (e.g., be discovered atstep 211).

At step 291, the flow may terminate until another device becomesdiscoverable, cluster join requirements are modified (making non-memberdevices eligible to join the cluster), or the like.

FIG. 3A is a diagram 300 depicting the relationships and/or transitionsbetween cluster device operation modes, such as cluster master, backupmaster, and active operational modes.

When a device is joined to a cluster, the device may begin operating ina default operational mode 310. As discussed above, if the device is thefirst device to join the cluster, the default operational mode 310 ofthe device may be the cluster master operational mode 320. If the devicejoins a cluster that already has a cluster master, but not backupmaster, the default operational mode 310 of the device may transition tobe the backup master 330. If the cluster already includes devicesoperating as cluster master 320 and backup master 330, the defaultoperational mode 310 of the device may transition to one of the active340 or standby 350 modes.

A device may operate as an active cluster device 340 if the cluster caninclude additional active (worker) devices (e.g., according to thelicensed capabilities of the cluster 300). The number of active clusterdevices may be defined by a cluster configuration and/or licensinginformation (e.g., the configuration and/or license may specify that thecluster may include five active cluster devices). The number of activecluster devices allowed in the cluster may or may not include thecluster master 320 and/or backup master 330. If the cluster may acceptadditional active cluster devices, the device may transition from thedefault mode 310 to the active mode 340, in which the device may accepttasks from the cluster master 320. If the cluster already includes themaximum number of active cluster devices 340 and/or if the configurationdata specifies that a certain proportion of the devices in the clusterbe allocated to high-availability (standby mode 350), the device maytransition to the standby mode 350. As discussed above, a device instandby mode 350 may not actively perform processing tasks assigned bythe cluster master, but may actively synchronize cluster configurationdata. Accordingly, when the device transitions from standby mode 350 toactive mode 340 (e.g., due to a change in cluster configuration,licensing, device failure, etc.), the device may be ready to beginperforming tasks assigned thereto without first synchronizing clusterconfiguration data. Other changes in the cluster configuration mayrequire that one or more active devices 340 transition back to standbymode 350. The transition may include the devices continuing tosynchronize cluster configuration data, but not accepting processingtasks (network flows) from the cluster master 320.

If the device operating as the cluster master 320 is demoted, anotherdevice may become (or be “elected” as) the cluster master 320. A devicemay be demoted from cluster master 320 for a number of different reasonsincluding, but not limited to: device failure (hardware, software,communications interface, or the like), device health score fallingbelow a threshold, configuration message from a human operator,automatic demotion (e.g., as a result of a failure detected within thedevice by the cluster master device or another monitoring device), orthe like.

When the cluster master 320 is demoted, a failover operation may occur.Failover may comprise promoting another device to operate as the clustermaster 340. If the cluster includes a backup master device 330, thebackup master device 330 may be elected as the new cluster master 320.Promoting the backup master 330 to the master 320 may includeconfiguring the other devices in the cluster (e.g., the active devices340 and/or demoted cluster master 320) to use the backup master device330 as the new cluster master). Since the backup master 330 may besynchronized to the cluster master 320 (may have been receiving updatesto the global, run-time synchronization data from the cluster master 32,such as flow assignment data, shared resource, data, session data,security data, and the like), the transition to the new cluster master320 may be performed without incurring downtime and/or interrupting theservices provided by the cluster.

In some embodiments, the backup master 330 may only be elected to thecluster master 320 operational mode if it satisfies some electioncriterion, which may relate to a minimum health score of the device,device hardware capabilities, processing load, or the like. If thebackup master 330 does not satisfy these criteria, and another clusterdevice does, another device other than the backup master 330 may beelected to operate as the cluster master 320. The election may comprisethe backup master transmitting the global, run-time synchronization datamaintained thereby to the new device. The performance penalty sufferedby transmitting the global, run-time synchronization data to the newcluster master may be mitigated by the fact that a device better suitedto act as the cluster master is put into place (e.g., reducing thechance of another failure in the short term). Alternatively, or inaddition, if the backup master 330 is deemed to be unsuitable to act asthe cluster master (and other cluster device is selected instead), thebackup master 330 may act as the cluster master 320 for a “transitionperiod,” until the global, run-time synchronization data is transmittedto the more suitable device, after which the more suitable device maytransition to cluster master 320, and the backup master may resume itsformer role.

Transitioning the backup master 330 to operate as the cluster master 320may include electing another device in the cluster to operate as thebackup master 330. If another device is available to act as a backupmaster 330, the device may be configured to transition to the backupmaster 330. The transition of a cluster device to backup master 330 maycomprise transmitting the global, run-time synchronization data to thenew backup master 330 (from the former backup master 330 or the failedover cluster master 320). In some embodiments, electing a new backupmaster 330 may comprise determining which, if any, cluster devices 340or 350 are eligible to operate in the backup master operational mode 330(e.g., based upon health score, processing load, device capabilities,such as processor speed, storage space, number and/or type of availablecommunications interfaces, and the like). If more than one clusterdevices are eligible for promotion to backup master 330, the electionmay comprise selecting the device with the higher health score, lower IPaddress, lower port number, or the like.

If no backup master 330 is available to replace the demoted clustermaster 320, a new cluster master 320 may be selected from the activecluster devices 340. The election may operate as described above (e.g.,based on device capabilities, health score, load, port number, or thelike). Following the election of the new cluster master 320, a newbackup master 330 may be elected as described above.

FIG. 3B depicts data flow 301 between cluster devices. As discussedabove, run-time synchronization data for load sharing and/or failovertransparency may be synchronized between cluster devices. In someembodiments, the cluster master 320 may be configured to synchronizecluster configuration (e.g., cluster configuration updates), flow,run-time synchronization data, shared source data, and the like to thecluster devices (e.g., the backup master 330, active device(s) 340,and/or standby device(s) 350). As shown in FIG. 3B, the cluster master320 may receive configuration updates (e.g., from a human operator via aconfiguration interface, from a policy server, or the like). Theconfiguration updates may include modifications to the cluster policy.The cluster master 320 may synchronize updates to the cluster policy tothe cluster devices 330, 340, and/or 350. Synchronizing the clusterpolicy may include modifying an operational mode of one or more clusterdevices (e.g., transitioning devices operating in standby 350 to activemode 340, or the like).

The cluster master 320 may be configured synchronize the global,run-time synchronization data with the backup master 330. As discussedabove, the global, run-time synchronization data may include data neededfor cluster master failover transparency, such as flow assignmentinformation (e.g., flow assignment data structure), flow, run-timesynchronization data, shared resource information (e.g., address pools,port pools, and so on), shared services information (e.g., securitykeys, security associations, etc.), and the like.

In some embodiments, the backup master 330 may be configured to transmitan acknowledgement message to the cluster master 320 responsive toreceiving global, run-time synchronization data therefrom. Theacknowledgement may be used by the cluster master 320 to verify that theglobal, run-time synchronization data was received. If the clustermaster 320 does not receive an acknowledgement from the backup master330 within a threshold period of time, the global, run-timesynchronization data may be retransmitted to the backup master, and/or anew backup master 330 may be elected as described above (a backup masterfailover operation may be performed). The global, run-timesynchronization data, cluster configuration, and other cluster stateinformation (e.g., key negotiation requests, etc.) distributed via thedevice cluster interface ports (e.g., ports 111A-111C of FIG. 1).

The cluster master 320 may be configured to distribute network trafficand/or flow processing tasks to the devices, including the activedevices 340, the cluster master 320 itself, and/or the backup master 330(the cluster master 320 and the backup master 330 may be used as“active” cluster devices for the purposes of flow processing). Thecluster master 320 may maintain a data structure indicating which taskshave been assigned to which cluster members (a flow assignment datastructure described below). The flow assignment data structure mayinclude an enumeration of the network flows (e.g., network connections,VPN connection, etc), being serviced by the cluster and identify whichdevice is servicing which flow. The cluster master may, therefore,monitor which devices are heavily loaded and which are less loaded andmake task assignment decisions accordingly. As will be discussed below,the cluster configuration may define one or more flow assignment rules,which may specify which devices are eligible to handle which flows(e.g., based upon existing flow assignments, security group information,session state, efficiency considerations, or the like).

The data sent between the cluster devices (e.g., cluster configurationdata, cluster state synchronization data, etc.) may be transmitted usinga secure communications channel. In some embodiments, data may beencrypted and/or digitally signed. As discussed above, inter-clustercommunications may be implemented on a cluster interface port (port111A-111C of FIG. 1). The cluster interface ports may implement ahigh-performance protocol that is resistant to application-layerfailures. One example of a high-performance, low-level communicationsprotocol is described below.

The active cluster devices 340 and/or backup master 330 may beconfigured to transmit flow, run-time synchronization data to thecluster master 320. The flow, run-time synchronization data may includedata relating to the flow(s) handled by the respective device(s).Accordingly, the flow run-time synchronization data may include all thedata needed to transition a flow from one cluster device to anothercluster device in the event of a failover operation. The flow, run-timesynchronization data may include flow session data, security information(e.g., security association sequence information number, shared keys,etc.), flow termination, addition and/or removal of rules on a datachannel, flow port assignments, flow cache, and the like.

The cluster master 320 may aggregate the flow, run-time synchronizationinformation received from the cluster devices into a global, run-timesynchronization data structure, which may be synchronized with thebackup master 330. As will be discussed below, when a device handling aparticular set of flows is failed over, the flows may be transitioned toone or more other cluster devices. The flow run-time synchronizationdata corresponding to each of the transitioned flows may allow thereplacement cluster device to resume processing the flows with minimalinterruption of service. In addition, the cluster master 320 maysynchronize the global, run-time synchronization data (including theflow, run-time synchronization data of each of the flows), to the backupmaster 330 to provide protection in the event of a failover operation ofthe cluster master 320 (e.g., in the event that the cluster master 320goes down, to be replaced by the backup master 330 or some other clusterdevice).

The global, run-time synchronization data may include other types ofsynchronization data, such as synchronization data pertaining to sharedresources managed by the cluster master 320, shared services provided bythe cluster master 320, cluster configuration data, and the like. Forinstance, in some embodiments, the cluster master 320 may manage IPsecurity (IPSec) data across the cluster. Accordingly, the clustermaster 320 may implement an Internet Key Exchange (IKE) module, whichmay provide shared IKE services to the other devices in the cluster (oneexample of such a configuration is described below in conjunction withFIG. 9A). When the cluster master 320 is configured to provide a sharedIKE, the cluster master 320 may provide call backs for use by the othercluster devices in the negotiation of security associations (e.g., phase1 security associations, phase 2 security associations, etc.), performdead peer detection, terminate IPSec flows, and the like. The global,run-time synchronization data synchronized from the cluster master 320to the backup master 330 may include the IKE module data to provide forIKE module failover.

The cluster master may manage the shared resources of the cluster.Shared resource information may be maintained within the global,run-time synchronization data structure discussed above (e.g., alongwith the flow assignment data structure, run-time flow synchronizationdata, and other cluster synchronization data managed by the clustermaster). The shared resources managed by the cluster master may include,but are not limited to: shared address pools, port pools, hostouttraffic configuration, and the like. In some embodiments, the clustermaster may act as a Dynamic Host Configuration Protocol (DHCP) server todynamically assign IP addresses to DHCP and/or to MOVPN clients (e.g.,IPSec, SSL, PPTP, and the like). The cluster master 320 may maintain anaddress pool data structure indicating which addresses have beenassigned to which clients. The address pool data structure may beincluded in the global, run-time synchronization data that issynchronized from the cluster master 320 to the backup master 330.

The cluster master may also manage a port pool for DNAT. When DNAT isused, the source IP address of a network flow may be replaced with afixed IP address, while a source port thereof is replaced with a portnumber obtained from a managed port pool. The assigned port number maybe reserved for use by the corresponding flow (e.g., the port may not beused for other network traffic, such as other network flows, while portis in use by the assigned flow). Hence, the cluster master may maintaina port pool comprising a data structure indicating which ports have beenassigned to which flows. The port pool information may be included inthe global, run-time synchronization data synchronized from the clustermaster 320 to the backup master 330.

The cluster master 320 may manage hostout traffic, which may includenetwork traffic that is initiated by cluster devices (individual clustermembers 320, 330, 340, and/or 350). The hostout traffic may be used forvarious purposes including, but not limited to: sending data to aquarantine server (e.g., in virus scanning, spam filtering, or thelike), sending logging information to a log server, sending SimpleNetwork Management Protocol (SNMP) traps to an SNMP manager, and thelike. The hostout traffic may use the shared cluster address (thecluster IP address) as the source of the traffic. Since the clusteraddress is shared, each cluster device that initiates hostoutcommunications may be assigned a different port, to prevent portconflicts between devices. The cluster master 320 may maintain a hostoutdata structure indicating which hostout ports have been assigned towhich cluster devices. The hostout data structure may be included in theglobal, run-time synchronization data that the cluster master 320synchronizes with the backup master 330.

The cluster master 320 may also manage shared resources associated withnetwork flows. For example, certain network flows may require the use ofparticular ports. For example, when an FTP session receives a “port”command or passive response the cluster device handling the flow may berequired to request of port number for the FTP session from the clustermaster 320. The cluster master 320 may use the port pool (or other datastructure) to determine which ports are available for use by the flowand/or to prevent a port conflict between network flows being handled ondifferent cluster devices. The cluster master 320 may use the port pool(discussed above), or another data structure, to prevent port conflicts.The data structure may include in the global, run-time synchronizationdata, which may be synchronized to the backup master 330 by the clustermaster 320.

Although the disclosure provides various examples of global, run-timesynchronization data that may be synchronized within the cluster of FIG.3B, the disclosure is not limited in this regard. The global, run-timesynchronization data described herein could include any type of dataand/or data structure known in the art. Similarly, synchronization datatransmitted to the cluster master 320 from the cluster devices 330, 340,and/or 350 (e.g., flow, run-time synchronization data) could include anydata and/or data structure known in the art.

In some embodiments, the cluster devices (e.g., the cluster master 320,backup master 330, worker devices 340, and/or standby devices 350) maybe configured to monitor the operational status of one another.Responsive to the monitoring, the cluster device(s) may take one ofseveral possible actions including, but not limited to: replacing thecluster master 320 with another cluster device, replacing the backupmaster 330 with another device, failing over a device (e.g., replacingthe device with another cluster device), or the like.

The cluster devices may implement a monitoring function in variousdifferent ways. In one example, each of the cluster devices may generateand transmit periodic status messages. The status messages may betransmitted using a shared cluster interface (e.g., interface ports111A-C of FIG. 1). The status messages may provide an indication thatthe cluster device is operational and capable of accepting tasks fromthe cluster master 320.

In some embodiments, the status messages may be used to communicatedevice performance and/or operational metrics, which may be used tocalculate or derive a “health score”, of the device. As used herein, adevice health score may refer to data (e.g., embodied as one or morealpha numeric values, formatted data, or the like), which be indicativeof the operational status of a cluster device. Accordingly, a healthscore may include, but is not limited to, providing indications of thestatus of one or more application layer modules implemented on thedevice; providing indications of the performance of the cluster device;providing indications of the status of the communications interfaces ofthe device; and the like.

For example, a device health score may include application-layerinformation, such as performance metrics of various deviceapplication-layer modules (e.g., traffic processing modules, etc.), thenumber of packets processed by the application per unit time,application throughput, may provide a record of application-layer faultsand/or application-layer exceptions, provide application resource usagemetrics (e.g., memory usage, processor usage, etc.), and the like. Theapplication-layer information in the health score may be provided on aper-application basis and/or as an aggregate of all application-layermodules. Other health score metrics may quantify the overall performanceof the cluster device, such as network throughput, overall processingload, system resource status, and the like. Health score metricsindicate the status of the communications interfaces of a clusterdevice. For example, the health score may indicate which (if any)communications interfaces are down (e.g., interface link status),provide interface-specific signal-to-noise ratio(s), provide indicationsof interface collision(s), provide indications of communicationinterface load, and the like.

In some embodiments, certain components of the health score may be moreheavily weighted than others. The weighting may allow an administratorto configure the health score to emphasize certain aspects of deviceperformance and/or health. For example, if a certain application orfunction is considered to be particular important to an organization,the administrator may configure the heath score to give added weight todevice metrics relating to the application and/or function.

In some embodiments, the health score of a device may include a failoverrequest. For example, the device may be due for maintenance (e.g., asoftware or hardware updates). Responsive to the maintenancerequirement, the device may transmit a status message comprising ahealth score (or other data), indicating that the device is to be takendown. Similarly, when the device determines that it is no longerproviding acceptable levels or service (e.g., due to application-layerfailures, such as VPN failure, anti-virus failure, or the like), thedevice may transmit a status message comprising a request for failover.

The status messages of the cluster devices may be used to determinewhich cluster devices should be selected to perform particularprocessing tasks, elect cluster devices to perform different roleswithin the cluster (e.g., cluster master 20, backup master 330, worker340, etc.), provide a quantitative gauge of the performance of thecluster device, provide an indication of the stability of the device,provide an operational status of the device (e.g., indicate whichservices or tasks the device is capable of performing), provide anindication as to whether the device is likely to fail within aparticular time frame, and so on.

As discussed above, in some embodiments, a health score (or data fromwhich a health score may be derived) may be included in the periodicstatus messages transmitted from a cluster device to the other clustermembers (e.g., via the cluster interface port 111A-C of FIG. 1). Thestatus messages (and/or the health score data therein) may be used tomonitor the cluster devices, which may comprise: performing failoveroperations, assigning processing tasks (network flows) to variouscluster devices, electing devices to act as the cluster master 320and/or backup master 330, and so on.

For example, in some embodiments, the health score of a device may beused to detect device instability which, may be indicative that thedevice is about to fail and/or is not operating properly. If the healthscore indicates that a device is about to fail, other cluster devicesmay implement a preemptive failover operation to failover the devicebefore it crashes. A preemptive failover may provide for a moreefficient transition to a replacement device, which may minimizeinterruption to the services provided by the cluster. Similarly, thehealth score of a cluster master 320 or backup master 330 may be used toinvoke a preemptive failover to replacement cluster master 320 and/orbackup master 330 devices.

The selection of a replacement cluster master 320 and/or backup master330 may be predicated upon, inter alia, the respective health scores ofthe available cluster devices (e.g., a cluster device having a lowhealth score may be excluded from election as the cluster master 320 orbackup master 330).

Failover of an active cluster device (e.g., the cluster master 320,backup master 330, or worker 340) may occur responsive to one or morefailover conditions including, but not limited to: loss of communicationwith the device (e.g., based upon device link status), communicationsinterface failures (e.g., failures in one or more non-clustercommunications interfaces), device crash (e.g., non responsive despitethe existence of a communications link), heath score below a threshold,in response to a configuration command (e.g., a command to bring downthe device to perform device maintenance, device upgrades, or the like),and so on.

Upon detecting failover conditions in a particular cluster device, theother devices in the cluster may implement a failover operation toreplace the failed over device with another device. The nature of afailover operation may vary according to the operational role of thefailed device.

For example, a cluster master failover may comprise selecting a newcluster master 320 from the other cluster devices. The selection of thenew cluster may be made by the remaining cluster members as a whole toensure that only one device is configured to act as the cluster master320 at any one time. The selection of a replacement cluster master 320may be based on the current role of each of the remaining clusterdevices. For example, if the cluster includes a backup master 330, thebackup master 330 device may transition to be the new cluster master 320(since it is already synchronized with the cluster master). If there isno backup master 330, or the backup master device also satisfiesfailover conditions (has failed or is on the verge of failing), a worker340 or standby 350 device may be selected. The selection may be basedupon device health score, processing load, IP address, connection speed,random selection, or the like. If the backup master 330 is operable (butnot suitable as the cluster master 320), the backup master 330 may beconfigured to synchronize the global, run-time synchronization data tothe replacement cluster master before being failed over itself.

If the device to be failed over is the backup master 330, a suitablereplacement may be selected as described above. The failover to areplacement backup master 330 may comprise synchronizing the global,run-time synchronization to the new device from the cluster master 320and/or backup master 330, if possible.

The device to the failed over may be actively performing clusterprocessing tasks (e.g., handling network flows). A failover operationmay comprise transitioning network flows assigned to the failed deviceto one or more available cluster devices. Transitioning a network from afirst device (failed device) to a second replacement device may comprisetransmitting to the second replacement device any flow, run-timesynchronization data pertaining to the flow. The flow, run-timesynchronization data may be transmitted to the second replacement deviceby the cluster master 320 or backup master 330, both of which maymaintain synchronized global, run-time synchronization data structurescomprising the flow, run-time synchronization data pertaining to theflows. The transition may further comprise the cluster master 320configuring the second replacement to handle the transitioned networkflows, updating the flow assignment data structure within the global,run-time synchronization data, and (if operating in “direct-forward”mode) configuring an inbound network interface to forward networktraffic associated with the flow to the second replacement device.

FIG. 3C is a flow diagram of one embodiment of a method 302 formonitoring a cluster using devices within the cluster. The method 301may be implemented on a computing device that has joined a cluster as acluster master, backup master, active member, and/or standby member. Thecluster device implementing the method 302 may comprise a processor andmemory. The method 302 may be embodied on the cluster device as one ormore computer-readable and/or computer-executable instructions, whichmay be embodied as one or more distinct software modules stored on acomputer-readable storage medium of the cluster device (e.g., hard disc,optical storage media, memory, or the like). In some embodiments, one ormore steps of the method 302 may be tied to particular devicecomponents, such as computer-readable storage media, communicationsinterfaces, processors, or the like.

At step 311, the method 302 may be initialized, which may compriseloading one or more computer-readable instructions from one or morecomputer-readable storage media, accessing and/or initializing one ormore communications interfaces, and the like.

At step 321, data indicative of a health score of the deviceimplementing the method 302 may acquired. As discussed above, a healthscore may comprise information relating to the application layer of thedevice, device performance, device communications interface status, andthe like. In some embodiments, step 321 may comprise calculating one ormore alpha numeric health score values (e.g., a set of alpha numericvalues, each relating to a different device health category).Alternatively, or in addition, step 321 may comprise acquiring data fromwhich a health score of the device may be derived (e.g., raw performancestatistics, operational parameters, logging information, and the like).

At step 331, a status message may be transmitted to other devices in thecluster. The status message may comprise the data acquired at step 321(e.g., the health score and/or the data used to derive the healthscore). In some embodiments, the status message may be transmitted via adedicated cluster communications interface port, such as the clusterinterfaces 111A-111C of FIG. 1. Alternatively, or in addition, thestatus message may be directed to a dedicated cluster network interface,such as the cluster interface 124 of FIG. 1, which may comprise aswitch, hub, concentrator, or other network communications device.

In some embodiments, the status message may be communicated using acluster-specific communications protocol, which may provide forefficient communications that are resistant to application-layerfailures. The cluster-specific communications protocol may beimplemented below the application layer (e.g., in the data link layer orthe like). The method 302 may be configured to generate and transmitstatus messages (e.g., perform steps 321 and 331) at regular intervals.Accordingly, other devices monitoring the device implementing the method302 may detect a failure in the device using the status messagestransmitted thereby and/or if no status messages from the device arereceived within a threshold time period.

At step 341, the method 302 may receive status messages from otherdevices in the cluster (e.g., via the dedicated cluster communicationsinterface, cluster network interface, or the like). The status messagesmay include respective device health scores (or information from whichrespective device health scores may be determined), each correspondingto a respective cluster device. In some embodiments, the status messagesmay further include information describing the current configuration ofthe device (e.g., cluster configuration, security policy, softwareversion, firmware version, etc.).

At step 351, the method 302 may determine whether any of the devices inthe cluster are to be failed over. A device may be failed over when oneor more failover conditions are satisfied. The method 351 may implementany number of different failover conditions, some of which may be basedupon the health score of the device, and other which may be related tolower-level monitoring functions (e.g., link-level monitoring,connectivity, and the like). The failover conditions at step 351 mayinclude, but are not limited to: the health score of the device fallingbelow a threshold; the health score of the device being maintained belowa threshold for a threshold time period; run-away device resourceconsumption as indicated by the health score (e.g., runaway processorload, memory usage, or the like); poor application-layer performance(e.g., if the health score indicates that one or more applicationsdeemed to be critical (IPSec, VPN, anti-virus, etc.) are not performingadequately; device hardware failures (e.g., bad memory, disc, or thelike); communications interface failures or performance degradation(e.g., low network throughput, high SNR ration, etc.); failure toreceive status messages for a threshold time period; device link status;hardware or software configuration change; or the like.

If at step 351, one or more cluster devices are to be failed over, theflow may continue at step 361; otherwise, the method 302 may terminateat step 371 and/or continue as more status messages are received and/orwhen an updated status message is to be transmitted.

At step 361, a failover operation for the one or more devices identifiedat step 351 may be performed as described above. An example of a methodfor failing over a cluster device is described below in conjunction withFIG. 3D. After failing over the device, the method 302 may terminate atstep 371 and/or may continue as more status messages are received fromother cluster devices and/or when a periodic status message is to betransmitted from the device.

FIG. 3D is a flow diagram of one embodiment of a method 303 for failingover a cluster device. Like the method 302 described above, the method303 may be implemented on and/or in conjunction with a cluster devicecomprising a processor, memory, communications interfaces, and the like.The method 303 may be implemented on the cluster device using one ormore computer-readable instructions embodied as discrete softwaremodules stored on a computer-readable medium.

At step 312, the method 303 may be initialized, which may compriseloading one or more instructions from a computer-readable medium,initializing communications interfaces, and the like.

At step 322, a device to be failed over may be identified. Theidentification of step 314 may be implemented by a method, such asmethod 302 described above in conjunction with FIG. 3C. Alternatively,the identification may be received from the device to be failed over(e.g., as an explicit failover request), received from an externalsource, such as a configuration interface or other management device(e.g., an SNMP message, user request, or the like), or the like.

At step 332, one or more available cluster devices to replace the faileddevice may be selected. The selection may be based upon health score ofthe other devices, the availability of standby devices, or the like. Ifthe failed over device is operating as the cluster master, the selectionof step 332 may give preference to a backup master (if available) asdiscussed above. If the backup master is selected to replace the clustermaster, step 332 may further comprise selecting a replacement for thebackup master.

At step 342, the one or more devices selected at step 332 may beprepared to handle the tasks of the failed over device. Preparing adevice to handle a processing task may comprise transferring flow,run-time synchronization data to the device. For network flowprocessing, the flow, run-time synchronization data may comprise flowsession information, such as IPSec data (e.g., P1SA, P2SA, shared keys,etc), flow cache, flow port shared resource allocations (e.g., DNATports, etc.), and the like. The flow, run-time synchronization data maybe transferred from the cluster master and/or backup master. If neitherthe cluster master nor backup master is available and/or has the flow,run-time synchronization data, and one or more communications interfacesof the device to be failed over are active, the data may be transferredto the replacement devices directly from the failed over device (beforeit is brought down and/or removed from the cluster).

If at step 342, the device to be failed over is the cluster master, thepreparation of step 342 may further comprise synchronizing the global,run-time synchronization data maintained by the cluster master to thereplacement cluster master device. If the replacement cluster masterdevice was formerly operating as the backup master, the global, run-timesynchronization data may already have been synchronized. If not, thesynchronization may take place between the backup master (if available)and the replacement cluster master or between the replacement device andthe cluster master itself (if possible). If the backup master wasselected to replace the cluster master at step 332, the preparation ofstep 342 may further comprise preparing a replacement for the backupmaster. Preparing a replacement for the backup master may comprisesynchronizing the global, run-time synchronization data to thereplacement backup master device.

At step 352, the cluster and the one or more replacement devices may beconfigured to perform the tasks of the failed over device. Theconfiguration of step 352 may comprise updating a task assignment datastructure to indicate which devices are handling the tasks formerlyassigned to the failed over device. For example, a flow assignment datastructure (discussed below) may be updated to indicate that the flowsthat were formerly being handled by the failed over device are not beinghandled by one or more replacement devices. If the cluster is operatingin direct-forward mode, the configuration of step 352 may furthercomprise configuring an inbound network interface (e.g., interface 120of FIG. 1) to forward traffic pertaining to the transferred flows to thecorresponding replacement devices.

If the failed over device was the cluster master, the configuration ofstep 352 may comprise the replacement cluster master taking overmanagement of shared cluster resources (e.g., address pools, port pools,etc), performing task assignment (e.g., assigning network flows tovarious cluster devices), maintaining global, run-time synchronizationdata, managing shared services (e.g., shared security services, such asIKE, and the like), and so on. Step 352 may, therefore, compriseconfiguring the other devices in the cluster to use the replacementdevice as the cluster master. Accordingly, the devices may be configuredto transmit flow, run-time synchronization data to the new clustermaster, request shared resources from the new cluster master, accessshared services from the new cluster master, and so on.

At step 362, the method 303 may terminate until another failoveroperation is to be performed.

In some cases, a cluster failure may be accompanied by a clusterpartition, in which a first set of one or more cluster devices are cutoff from communication with a second set of cluster devices. Theelection of a cluster master may be configured to prevent “clusterpartitioning,” in which each set of communicatively coupled clusterpartitions elects its own cluster master (resulting in two concurrentlyrunning cluster masters 320). The cluster devices may detect a clusterpartition (e.g., split syndrome in which some cluster devices cannotcommunicate with other cluster devices). The detection may compriseusing a specially configured Ethernet frame to probe the multi-segmentcondition. If a multi-segment condition is detected, an active segmentmay be selected. The active segment may be the segment comprising thelargest number of computing devices, the segment comprising the clustermaster and/or backup master, or the like. When a device on an inactivesegment reconnects to an “active segment” device, the device may besynchronized thereto. For example, the cluster master in the activesegment may rejoin the device to the cluster as described above.

Referring back to FIG. 3B, the devices in the cluster 301 (clustermaster 320, backup master 330, worker device(s) 340, and/or standbydevice(s) 350) may synchronize cluster information using a dedicatedcluster interface port, such as the cluster interface ports 111A-111C ofFIG. 1 (e.g., each device may dedicate a communications interface tocluster communications, which may be concentrated at a switch or othernetwork element). In some embodiments, the cluster interface port mayimplement a specialized protocol configured to provide for high-speed,robust device-to-device communications. The protocol may operate at alow level to provide for cluster communication despite failures in theapplication layer of the device. For instance, the protocol may beimplemented with the OSI data layer.

Referring back to FIG. 1, the cluster 110 may be configured to providenetwork security services with load balancing and/or high availability.Load balancing may be implemented by assigning flows (comprising networkprocessing tasks) to cluster members as evenly as possible, while highavailability may be implemented by reassigning flows to survivingcluster members (or standby cluster members) when a cluster devicefails.

In some embodiments, the cluster 110 may implement a “flooding”technique, in which each arriving packet (received via the interface120) may be forwarded to each of the cluster devices 112, 114, and 116.Each cluster device 112, 114, and 116 may examine the packet header anddecide whether to handle the packet (e.g., based upon whether the devicehas been assigned to manage the flow associated with the packet). If so,the device may process the packet according to a policy implemented bythe cluster 110; otherwise, the device may drop (ignore) the packet. Ifthe packet is not associated with any known flow (e.g., a new inboundrequest, etc.), the cluster master may identify the flow as “new” andassign it to an active cluster device 112, 114, 116.

To operate in the flooding mode described above, the cluster masterdevice (device 112) may send an address resolution protocol (ARP) replyto the interface 120 with a multicast Media Access Control (MAC)address. A generic multiple registration protocol (GMRP) may be used toprevent flooding network traffic to ports not used by the cluster 110.Certain network elements (e.g., routers) may require an ARP entry forthe cluster master 110 to be manually inserted.

Alternatively, the interface master may send an ARP reply with anunknown unicast MAC address, which may cause the interface 120 to routeinbound traffic to all of the devices (112, 114, and 116) in the cluster110. However, some interface devices 120 may rate-limit trafficassociated with unknown destination MAC addresses. In another example,the interface 120 may include a dump hub. The dump hub may flood inboundnetwork traffic to the devices (112, 114, and 116) in the cluster 110.However, network traffic sent out by the cluster devices 112, 114,and/or 116 may loop back within the cluster 110 (e.g., traffictransmitted by device 112 may loop back to the devices 114 and 116, andso on).

In an alternative embodiment, the cluster 110 may be configured tooperate in a “direct-forwarding” mode. Direct forwarding may beimplemented using an interface 120 configured to route inbound networktraffic to a particular device 112, 114, or 116 within the cluster 110.In a direct forwarding scheme, the cluster master (e.g., device 112) mayregister a virtual MAC address with the interface 120 (e.g., respond toan ARP request (virtual IP) from the interface 120 with a virtual MACaddress). Therefore, unicast traffic will be routed to the clustermaster 112 only (and not to the other devices 114 and 116 in the cluster110). Multicast and/or broadcast traffic may be filtered by the devices(e.g., 114 and 116) that are not configured to act as the cluster master112. When a flow is assigned to a particular device 112, 114, or 116 inthe cluster 110, it may transmit a direct forward request to theinterface 120. The direct forward request may specify the types oftraffic that are to be forwarded to the device (e.g., in a trafficspecification) and provide the MAC address of the device (112, 114, or116) to which the traffic is to be forwarded. Using the information inthe direct forward request, the interface 120 may identify trafficassociated with the flow and forward the traffic directly to the device(as opposed to flooding each device 112, 114, and 116 therewith). If noflow is associated with the incoming traffic, the traffic may beforwarded to the cluster master, which may assign to flow to a clusterdevice 112, 114, 116.

FIG. 4 is a block diagram of one embodiment of a cluster device, such as112, 114, and/or 116. The cluster device 400 may be implemented asand/or in conjunction with a computing-device 410 comprising a processor412, memory storage 414, and computer-readable storage media 416. Theprocessor 412 may comprise one or more general purpose processors (e.g.,Intel® Pentium® processor(s), Advanced Micro Devices Athlon®processor(s), or the like), one or more special purpose processors, oneor more application specific integrated circuits (ASICs), or the like.The memory 412 may comprise volatile and/or non-volatile memory. Thecomputer-readable storage media 416 may comprise one or more hard discs,optical storage media, Flash storage media, and the like.

The device 400 may include a communications interface 450, which maycommunicatively couple the device to one or more networks, such as thenetwork 140 of FIG. 1, the Internet, a WAN, a LAN, a local-clusternetwork, or the like. The communications interface 450 may comprisewired and/or wireless communications interfaces, such as Ethernetinterfaces, fiber-optic interfaces, IEEE 802.11 interfaces, and thelike. One or more of the communications interfaces 452 may becommunicatively coupled to a communications network 440, which maycomprise a WAN and/or the Internet. In some embodiments, thecommunication interface(s) 120A may be communicatively coupled to thecommunications network via a cluster interface 420.

One or more communications interfaces 454 may be communicatively coupledto an internal network 430 (e.g., LAN, local network, home network, orthe like), such as the organization network 130 of FIG. 1. Thecommunications interface(s) 454 may be communicatively coupled to theinternal network 430 via a communications interface 122, which maycomprise a switch, router, or other network element.

One or more communications interfaces 456 may be communicatively coupledto a local cluster network, which may provide for communications withother cluster devices (not shown), such as the devices 112, 114, and 116in the cluster 110 of FIG. 1. Accordingly, the communications interfaces456 may be communicatively coupled to a switch, hub, router, or otherconcentrator device 424.

The device 410 may include one or more processing modules 460, 462, 464and 466, which may be operable on the processor 412 and/or implemented(in whole or in part) using one or more special purpose processingelements (e.g., special purpose processors, ASICs, or the like).Portions of the modules 460, 462, 464, and/or 466 may be implemented onthe processor 412 using one or more computer-readable instructionsstored on the computer-readable storage media 416.

A flow assignment module 460 may be configured to assign network flowsto the devices in the cluster (e.g., using a method, such as method 500described below). For example, when operating as a cluster master, thedevice 410 may receive inbound network traffic from the interface 420.The traffic may be routed to the flow assignment module 450, which mayidentify a flow associated therewith.

The traffic processing module 462 may process network traffic accordingto a security policy and a local, flow assignment data structure 463.The security policy enforced by the traffic processing module 462 may bedefined in the cluster policy data structure 470, and may specify, interalia, how various types of network traffic and/or network flows are tobe processed (e.g., allowed or not allowed, filtered, routed, and soon). For example, a security policy may determine which types of networktraffic may be passed from an external network (e.g., coupled tointerface 420) to an internal, organization network (e.g., coupled tointerface 422), and vice versa, may define network filtering tasks,define firewall rules, and so on.

In some embodiments, the network security policy defined in the clusterpolicy data structure 470 may define a role-based, user-based, or othertype of network security policies. For example, the clusterconfiguration 470 may reference and/or provide a link to a networkauthentication or authorization server (not shown), which may be used toprovide user- and role-based security services. Accordingly, in someembodiments, the traffic processing module 462 may be communicativelycoupled to one or more external servers (not shown), from which securitypolicy information may be obtained. Alternatively, or in addition, suchsecurity policy information may be cached within the clusterconfiguration 470 and/or updated by the device acting as the clustermaster.

The traffic processing module 462 may maintain a local, flow assignmentdata structure 463 identifying the network flows that have been assignedthereto. The identifying may comprise information to allow the trafficprocessing module 462 to associate network traffic with an assigned flow(e.g., based upon source address, destination address, protocol, port,security information, and so on). Unlike the flow assignment datastructure 472 maintained by a cluster master that provides informationregarding all the network flows handled throughout a cluster, the datastructure 463 may identify only the flows that have been assigned to theparticular device 410. When the cluster master assigns a flow to thedevice 410, the traffic processing module 462 may update the local, flowassignment data structure 463. The data structure 463 may also be usedto manage local, run-time synchronization data associated with the flowsassigned to the device 410.

The cluster configuration data 470 may also include informationidentifying each of the devices within the cluster, identifying thestatic roles of each of the cluster devices (e.g., active, standby,etc.), indicate a current state of each of the cluster devices (e.g.,active, standby, failed, etc.), provide cluster licensing information,and so on. As will be discussed below, when the device 410 is operatingas the cluster master, the device 410 may be configured to synchronizethe cluster configuration data 470 to the other cluster devices.Therefore, the cluster master (device 410) may act as the “source” ofcluster configuration data for the other cluster devices. When changesto the cluster configuration are made (e.g., via a configurationinterface, policy server, or the like), the cluster master (device 410)may be configured to synchronize the changes to the other clusterdevices (e.g., cause the other cluster devices to update theirrespective cluster configuration data structures 470).

When operating as the cluster master, the device 410 may be responsiblefor managing the workload of the cluster. Accordingly, the clustermaster (device 410) may be configured to assign processing tasks, suchas handling network flows, to various active cluster members. Thecluster master device 410 may maintain a flow assignment data structure472, which may provide a mapping between the network flows being handledby the cluster, and the cluster devices assigned thereto.

When the cluster master (device 410) receives network traffic that isnot associated with a known flow and/or is associated with a flow thatis not being actively handled by a cluster device (according to the flowassignment data structure 472), the flow assignment module 460 mayassign the flow to a cluster device. Assigning a flow to a clusterdevice may comprise selecting a cluster device according to a set offlow assignment rules and/or other criteria (e.g., device health, deviceavailability, etc.) After selecting a cluster device to handle aparticular flow, the flow assignment module 460 may update the flowassignment data structure 472 accordingly. The cluster master may alsoconfigure the selected cluster device to begin processing the flow(e.g., transmit a message via a cluster communication interface 456 toconfigure the selected device to begin processing the network flow). Inaddition, when the cluster is operating in direct-forward mode, the flowassignment module 460 (or the device assigned to handle the flow) mayconfigure the interface 420 to route traffic associated with the flowdirectly to the device assigned thereto.

The cluster devices that are actively processing network flows may beconfigured to transmit flow, run-time synchronization data to thecluster master. The cluster master (device 410) may aggregate the flow,run-time synchronization data into data structure 474 comprising all ofthe flow, run-time synchronization of all the cluster devices. The flow,run-time synchronization data may include information needed to handlethe corresponding network flow (e.g., session information, stateinformation, security keys, sequence number, etc.). The flow, run-timesynchronization data may be transmitted to the device 410 via thecluster communication interface(s) 456 or another interface 452 or 454.When acting as a cluster master, the device 410 may be configured tomaintain the flow, run-time synchronization data to provide for flowfailover between cluster devices. As discussed above, when a clusterdevice fails, the flows assigned thereto may be transitioned to anotherreplacement cluster device. In the transition, the flow, run-timesynchronization data corresponding to the transitioning flows may betransmitted to the replacement device, allowing the device to resumehandling the flow with minimal disruption.

As discussed above, when a cluster device 410 is acting as a clustermaster, the device 410 may manage shared cluster resources, such asaddress pools, port pools, and the like. The cluster master (device 410)may also provide shared services, such as a shared IKE module. Theshared IKE module may provide callbacks to other cluster devices tonegotiate security associations (P1SA, P2SA, and so on), shared keys,and the like. Information relating to the shared resources and/or sharedservices provided by the device 410 may be maintained in a sharedresource data structure 476.

The cluster configuration data structure 470, flow assignment datastructure 472, flow, run-time synchronization data structure 474, sharedresource data structure 476, and any other data needed for clustermaster failover, may be maintained in a global, run-time synchronizationdata structure 480. When operating as a cluster master, the device 410may synchronize the global, run-time synchronization data structure 480with other cluster devices (e.g., the backup master device).Synchronizing the data structure 480 may provide for replacement of thecluster master by another cluster device (e.g., the backup master device(not shown)), in the event of a failover.

Accordingly, when the device 410 is operating as a backup master, thedevice 410 may be configured to receive the global, run-timesynchronization data structure 480 (comprising cluster configuration470, the flow assignment data structure 472, the flow, run-timesynchronization data 474, the shared resource data structure 476, andany other relevant data) from the cluster master.

The cluster master (device 410) may be configured to synchronize thecluster configuration data structure 470 to the other cluster devices.The traffic processing module 462 may use the cluster configuration datastructure 470 to process network flows in accordance with the securitypolicy defined therein. However, cluster devices other than the clustermaster and backup master may not actively use the flow assignment module460 and/or may not maintain the flow assignment data structure 472,flow, run-time synchronization data structure 474, and/or sharedresource data structure 480, since these structures are not needed forflow processing. In some embodiments, however, the modules (460, 464,and 466) and data structures (472, 474, and 476) may exist in skeletonform to provide for an efficient transition to a cluster master and/orbackup master role within the cluster when needed.

A monitoring and failover module 464 may be used to monitor clusterdevices as described above in conjunction with FIGS. 3C and 3D. Eachcluster device 464, whether operating in the cluster master, backupmaster, active, or standby role, may implement the monitoring andfailover module 464. The monitoring and failover module 464 may beconfigured to generate and transmit status messages from the device 410,which, as discussed above, may comprise a health score of the device.The module 464 may also be configured to receive status messages fromother cluster devices. Information relating to the health score of thedevice 410, as well as status information relating to other clusterdevices may be maintained in a monitoring and failover data structure478.

When operating as the cluster master, the device 410 may be configuredto provide for device failover within the cluster. For example, when adevice handling a particular set of flows fails, the processing tasksassigned thereto may be transitioned to other cluster devices. Themonitoring and failover module 464 may be configured to determine when afailover operation is needed (according to failover criteria, and asdiscussed above in conjunction with FIG. 3C). When a device for failoverhas been identified, the monitoring and failover module 464, along withthe flow assignment module 460, may assign the tasks of the failed overdevice to one or more replacement devices (as described above inconjunction with FIG. 3D). The flows may be transitioned to existing,active cluster devices and/or to standby cluster devices that have beenactivated responsive to the failure. When the new device(s) areselected, the flow, run-time synchronization data 474 associatedtherewith may be sent to the corresponding replacement devices, whichmay use the data to resume handling the flows.

If the device being failed over is operating as the backup master,failover may additionally include selecting a new device to act as thebackup master (e.g., based upon device health score, address, or thelike) as described above. After the backup master is selected, thecluster master may be configured to synchronize the global, run-timesynchronization data structure 480 to the new backup master device.

If the device being failed over is operating as the cluster master, anew cluster master may be selected as described above (e.g., as thebackup master, based upon health score, or the like). The global,run-time synchronization data structure 480 may be populated from thebackup master and/or failed over cluster master (if available).

When operating as cluster master, the device 410 may comprise a clustermanagement module 466, which may be used to synchronize the clusterconfiguration data structure 470 to other cluster devices (not shown),join new devices to the cluster (as described above in conjunction withFIG. 2B), determine and maintain the licensed capabilities of thecluster, and so on.

As discussed above, when a new computing device (not shown) iscommunicatively coupled to a cluster network interface 420, 422, and/or424, the device may be discovered by other cluster devices. Discoverymay comprise a cluster device (e.g., the cluster master device 410)transmitting one or more discovery messages (e.g., broadcast, multicast,or other message types) on one or more of its communications interfaces450. The discovery messages may be sent automatically (e.g.,periodically, until a device is discovered). In some embodiments, theperiodic status messages transmitted by the monitoring and failovermodule 464 may be configured to also serve as discovery messages.Alternatively, the discovery messages may only be transmitted uponreceiving a discovery command (or other command). The configurationcommand may be received via one or more of the communications interfaces450 and/or a configuration interface 467 (discussed below). In someembodiments, the device 410 may discover a new computing devicepassively (e.g., by monitoring traffic on the interface 420, 422, and/or424, by inspecting routing and/or ARP information, or the like).

When a new computing device is discovered, the cluster manager module466 of the cluster master device 410 may be configured to initiate acluster join procedure to add the new device to the cluster as describedabove in conjunction with FIGS. 2A and 2B. The cluster management module466 may be configured to access device-identifying information of thenew computing device (e.g., by interrogation, passive monitoring, orother means). Using the device-identifying information, the clustermanagement module 466 may determine whether the discovered computingdevice is eligible to join the cluster (e.g., based upon hardware,software, firmware, licensing, or other information about the device).If the computing device is eligible to join the cluster (e.g., iscompatible with the other devices in the cluster, is licensed forcluster operation, and so on), the cluster management module 466 maydetermine a device-specific configuration for the new device, andtransmit the device-specific configuration thereto. The device-specificconfiguration may include the cluster configuration data 470, includingidentifiers of the cluster devices, cluster addressing information, andthe like. The device-specific configuration may also include a roleassignment specifying the static role of the new computing device in thecluster, provide device-specific addressing and port assignmentinformation, provide one or more device-specific shared keys for securecluster communications, and so on. After the transmission, the clustermanagement module 466 may verify that the new computing deviceimplemented the device-specific configuration (e.g., by a confirmationmessage, active interrogation, network inspection, or the like). If thenew computing device fails to implement the device-specific and/orcluster configuration, the new computing device may be excluded from thecluster.

When the cluster management module 466 verifies successfulimplementation of the device-specific and/or cluster configuration, thedevice may be joined to the cluster (e.g., added to the clusterconfiguration data 470, which is synchronized with other clusterdevices). Joining may comprise providing the new computing device with ashared key (or performing a key exchange protocol) to allow the deviceto securely communicate with other cluster devices. Upon joining thecluster, the new computing device may then begin performing its assignedrole (e.g., be assigned cluster processing tasks, receive clustersynchronization data, and the like).

The cluster management module 466 of a device 410 operating as thecluster master may also be configured to determine the licensedcapabilities of the cluster. Information defining the licensedcapabilities of the cluster may be included in the cluster configurationdata structure 470, which is synchronized to, and implemented by, theother cluster devices. In some embodiments, the licensed capabilities ofthe cluster may be defined in a single cluster license. Alternatively,the licensed capabilities of the cluster may be determined by combiningthe licenses of two or more cluster devices. As discussed above, thecombination may be made in a number of different ways, including, butnot limited to: a “least capabilities” combination, an additivecombination, a logical OR combination, a selective combination (e.g.,different combination types of for different licensed features), or thelike.

The cluster management module 466 may use the licensing information toassign static roles to cluster members (as devices are joined to thecluster, as the cluster configuration changes, and so on). For example,devices that do not have their own license may be assigned a static roleof “secondary” or standby. Devices that are appropriately licensed maybe eligible to be assigned a static role of “primary” or active. In someembodiments, the licensed capabilities of the cluster (defined in asingle cluster license, or by combining two or more cluster devicelicenses) may determine cluster configuration. For instance, if thelicensed capabilities of the cluster provide for five active devices,new computing devices may be added to the active role (up to five)regardless of their individual licenses. However, once five activedevices are in the cluster, additional devices may be assigned tostandby, even if the devices have individual cluster licenses.

In some embodiments, the cluster management module 466 may provide aconfiguration interface 467, which may comprise a network accessibleuser interface, an Application Programming Interface (API), an SNMPclient, a telnet server, a serial communications interface, a parallelport interface, or the like. Accordingly, in some embodiments, theconfiguration interface 467 may comprise and/or utilize one or more ofthe communications interfaces 450 (e.g., the communication interfaces454 communicatively coupled to an internal, or organization network).Alternatively, or in addition, the configuration interface 467 maycomprise one or more human-machine-interaction (HMI) components (notshown), such as a display, keyboard, mouse, or other input/outputdevices.

The configuration interface 467 may allow a human operator, policymanager, or other configurator to interrogate and/or change theconfiguration of the cluster. When the device 410 is operating as acluster master, the configuration interface 467 may be capable ofdisplaying and/or modifying the configuration of the cluster as a whole.Accordingly, the configuration interface 467 may be capable ofdisplaying the configuration of all of the computing devices in thecluster, including the health score and other performance indicatorsthereof, may be capable of setting configuration parameters of all ofthe computing devices in the cluster, and so on.

In some embodiments, the configuration interface 467 may implement a“single-device” paradigm, in which the cluster is viewed and managed asa single device. Accordingly, configuration changes made to one clusterdevice (the cluster master device 410), may be transparentlysynchronized to other cluster devices (by the cluster management module466). However, per-device configuration changes may be accessible byinterrogating individual cluster devices (e.g., to apply device-specificlicensing parameters, view device-specific information, such as healthscore and the like, and so on).

The cluster management module 466 (along with the configurationinterface 467) may provide for efficient cluster updating andmaintenance. For example, a command to upgrade or modify the clusterconfiguration may be received via the configuration interface 467. Theupgrade or modification may require that the devices in the cluster betaken down (e.g., may include a change to device software, firmware,and/or hardware). Responsive to such a request, the cluster managementmodule 466 may implement a cluster upgrade operation in which clusterdevices are taken down in an orderly fashion such that the servicesprovided by the cluster are not significantly impacted.

In one example, the cluster management module 466 may be configured toupgrade or modify cluster standby devices first. If the cluster includestwo or more standby devices, the upgrade or modification of may be donesuch that one or more of the standby devices is available during theupgrade. For example, if there are three standby devices A, B, and C,the cluster management module 466 may configure device A to be upgradedfirst, while keeping the B and C devices up and then, after the A deviceis running again, upgrade device B while devices A and C are kept up,and so on.

The cluster management module 466 may then perform a similar upgradeprocess on the active cluster devices; the cluster management module 466may be configured to upgrade the active cluster devices sequentially,taking down only one (or some other subset) of the active devices at atime. As each active device is upgraded or modified, it may be failedover to another active device and/or to a standby device if available.

The cluster management module 466 may then be configured to perform themodification to the cluster backup master device. The backup master maybe failed over to another cluster device before being modified (in afailover operation as described above). A replacement backup master maycontinue operating as the backup master even after the former backupmaster has been modified. The selection of the backup master may be madeaccording to a selection criteria as discussed above (e.g., based uponhealth score, resources, hardware capabilities, or the like).

After modifying the backup master, the cluster management module 466 maybe configured to modify the cluster master device itself. Modifying thecluster master device may comprise failing over the cluster master toanother cluster device (e.g., the replacement backup master or anothercluster device). After completing the cluster master failover operation(e.g., and selecting a new cluster master), the device 410 may beupgraded and rejoined to the cluster. The device 410 may resume actingas the cluster master (e.g., by failing over the temporary clustermaster selected above) and/or may resume operation in another rolewithin the cluster (e.g., as a backup master, active, or standby clusterdevice).

FIG. 5 is a flow diagram of one embodiment of a method for assigningprocessing tasks, such as network flow processing, in a cluster, such asthe cluster 110 of FIG. 1. The method 500 may be implemented on acomputing device, such as the device 410 of FIG. 4. The method 500 maybe implemented using one or more computer-readable and/orcomputer-executable instructions. The instructions comprising the method500 may be implemented as one or more distinct software modules, whichmay be stored on a computer-readable storage medium, such as a harddisc, optical storage media, memory, or the like. In some embodiments,one or more steps of the method 500 may be tied to particular machinecomponents, such as computer-readable storage media, communicationsinterfaces, processing modules, or the like.

At step 510, the method 500 may start and/or be initialized.Initializing the method 500 may comprise loading one or morecomputer-readable instructions from one or more computer-readablestorage media, accessing and/or initializing one or more communicationsinterfaces, and the like.

At step 520, network traffic may be received. The network traffic mayhave been received via a network interface (e.g., interface 120 and/or122 of FIG. 1), such as a hub, switch, router, concentrator or the like.If the cluster of method 500 is operating in “flooding” mode, thetraffic may be received by all of the devices in the cluster. If thecluster is operating in “direct forwarding” mode, the traffic may bereceived only by the cluster master device (or the device configured tohandle the flow).

At step 530, the cluster master may determine whether the networktraffic is associated with a known flow (e.g., a flow that is beinghandled by one of the devices in the cluster). Step 530 may compriselooking up the flow in a flow assignment data structure, such as atable, index, or the like. The flow assignment data structure mayprovide a mapping between network flows and the cluster devices assignedthereto. In the data structure, a flow may be identified based upon asource address thereof (IP address, MAC address, or the like), flowprotocol, flow port, flow security information, or the like. The clusterdevice assigned to the flow may be identified according to acluster-specific identifier, MAC address, cluster address (e.g., addressof a cluster interface port of the device), IP address, or the like. Insome embodiments, the flow assignment data structure may furthercomprise flow, run-time synchronization data pertaining to the flow,which may include, but is not limited to: flow security association data(P1SA, P2SA), shared key data, user-session data, flow cache, and thelike.

Alternatively, or in addition, the flow assignment data structure mayinclude a reference (link, pointer, or the like) to the flow, run-timesynchronization data associated therewith.

If at step 530, the method 500 determines that there is no deviceassigned to handle the flow (e.g., there is no entry for the flow in theflow assignment data structure, the device identifier associated withthe flow entry has not been set, the device that was handling the flowhas been failed over, or the like), the method 500 may continue at step540; otherwise, if a device is already actively handling the flow, themethod 500 may continue at step 535.

At step 535, since the network traffic is associated with a flow thathas already been assigned to a device in the cluster that is activelyhandling the flow (e.g., has not failed), the traffic may be ignored bythe method 500. Accordingly, the method 500 may terminate at step 560and/or may continue at step 520 when additional network traffic isreceived.

At step 540, the method 500 may select a cluster device to handle theflow. Selecting a cluster device may comprise identifying which devicesin the cluster are eligible to handle the flow (e.g., are active, basedupon flow assignment rules discussed below, and so on), evaluating aload and/or health of the devices in the cluster, and the like. In someembodiments, the cluster master and/or master backup devices may beavailable to handle network traffic flows. Alternatively, the clustermaster and/or backup master may be dedicated to managing the state ofthe cluster and not processing network traffic flows. Eligibility of acluster device to handle a particular flow may be based upon one or moreflow assignment rules (discussed below). The flow assignment rules maydetermine eligibility based upon whether another cluster device hasalready been assigned a network flow that is related to the new networkflow, shares security information with the new network flow, or thelike. If two or more cluster devices are eligible to handle the newnetwork flow, the selection of step 540 may comprise evaluating aselection criteria to select one of the two or more devices. Theselection criteria may be based upon device health score, processingload, random (e.g., round robin), or some other metric.

At step 550, the flow assignment data structure may be updated toreflect the assignment. At step 550, if the cluster is operating indirect forward mode, the cluster interface may be configured to forwardthe flow traffic directly to the device assigned to handle the flow(e.g., using a direct forward request or other configuration message).

At step 560, the flow may terminate and/or may continue at step 520 whenadditional network traffic is received.

As discussed above, assigning a flow to a device may comprisedetermining whether which devices are eligible to handle the flow and/orevaluating one or more flow assignment rules. A device may be eligibleto handle a flow if the health score of the device indicates thathandling the flow would not cause the device to become unstable, performpoorly, adversely affect quality of service (QoS) for other flowshandled thereon, or the like. Similarly, a flow assignment rule maydetermine which devices are eligible to handle a particular flow basedupon other flow assignments. For example, a flow assignment rule mayspecify that all flows that use the same tunnel go through the samecluster device. Consolidating flows in this manner may provide forprotection against anti-replay attacks and facilitate dead peerdetection (DPD).

FIG. 6A is a block diagram illustrating one example of related flowassignment in which related forward and reverse flows are assigned tothe same cluster device. In the FIG. 6A example, forward and reverseflows 681 and 682 are established between a computing device 633 withinan organization 630 and a computing device 646 in the network 640. Thecluster master device (device 612) may implement a flow assignment rulethat assigns related forward and reverse flows to the same clusterdevice. Accordingly, in the FIG. 6A example, both of the flows 681 and682 may be assigned to device 2 614. Alternatively, the flows 681 and682 could be assigned to another device 612, 614, or 616. The flowassignment rule may prevent the flows 681 and 682 from being assigned todifferent devices (e.g., flow 681 being assigned to device 614 and flow682 being assigned to device 616, and so on).

FIG. 6B is a block diagram depicting another example of related flowassignment in which related flows are assigned to the same clusterdevice. The related flows 681, 682, 683, and 684 depicted in FIG. 6B maybe related to one another (e.g., may be the data and control channels ofa file transfer protocol (FTP) connection, may be related to the samevoice over IP connection (VOIP), or the like). A flow assignment rulemay specify that all of the related flows are to be assigned to the samecluster device. As shown in FIG. 6B, related flows 681, 682, 683, and684 may be established between a computing device 633 in theorganization network 630 and an external computing device 646. Accordingto the flow assignment rule, all of the flows 681, 682, 683, and 684 mayall be assigned to the same device (device 2 614). Alternatively, theflows 681, 682, 683, and 684 may all be assigned to another device (612,616, or 618). The flow assignment rule may prevent the flows 681, 682,683, and/or 684 from being split up between different devices in thecluster (e.g., prevent flows 681 and 682 from being assigned to device 2614 and flows 683 and 684 being assigned to device 3 616, and so on).

FIG. 6C is a block diagram depicting an example of security flowassignment, in which flows associated with the same tunnel are assignedto the same cluster device. In the FIG. 6C example, a tunnel 680 (e.g.,an SSH tunnel, or the like) may be established between a computingdevice 633 in the organization network 630 and a computing device 646 inthe network 640. The tunnel 680 may comprise one or more flows 681, 682,683, and 684. The flow assignment rule may specify that all of the flows681, 682, 683, and 684 of the tunnel 680 are assigned to the samecluster device (device 2 614). Accordingly, the flow assignment rule mayprevent the tunnel 680 flows 681, 682, 683, and 684 from being split upbetween the devices 612, 614, 616, and/or 618.

FIG. 6D is a block diagram illustrating another example of security flowassignment in which flows associated with the same inbound or outboundsecurity association are assigned to the same cluster device, whereasthe inbound and/or outbound tunnel flows may be assigned to differentcluster devices. As shown in FIG. 6D, flows 681 and 682 use the sameoutbound SA 680 and flows 687 and 688 use the same inbound securityassociation. Accordingly, the flow assignment rule may specify that theflows 681 and 682 are assigned to the same device (device 2 614), andthe flows 687 and 688 are assigned to the same device (device 3 616).The flow assignment rule illustrated in FIG. 6D may prevent the flows681 and 682 from being assigned to different cluster devices and/orprevent the flows 687 and 688 from being assigned to different clusterdevices.

Alternatively, a flow assignment rule may specify that all of theinbound and outbound flows associated with a particular securityassociation be assigned to the same device. FIG. 6E shows an example ofthis type of flow assignment. As shown in FIG. 6E, the flows 681, 682,687 and 688 are all assigned to the same device (device 3 616). The flowassignment illustrated in FIG. 6E may prevent the flows 681, 682, 687and/or 688 from being split up across different cluster devices.

A cluster according to the teachings of this disclosure may beconfigured to provide tunnel switching (e.g., provide for communicationbetween two or more remote peers). When used for tunnel switching, thecluster master device may implement a flow assignment rule configured tospecify that the flows associated with the tunnel switch connection behandled by the same cluster device. A restriction rule of this type mayreduce cluster communication traffic (e.g., prevent tunnel switch datafrom being transferred between cluster devices). FIG. 6F is a blockdiagram that illustrates another example of a related flow assignment inwhich the flows associated with the same tunnel switch are assigned tothe same device. In the FIG. 6F example, remote peer devices 646 and 647establish a tunnel switch with the cluster 610. The flows 681 and 682may be associated with the remote peer 646, and the flows 687 and 688may be associated with the remote peer 647. The flows 681, 682, 683, and684 may be used to implement a tunnel switch between the remove peers646 and 647 (e.g., provide for peer-to-peer communication therebetween).The cluster master device 612 may identify the flows 681, 682, 687, and688 as forming a tunnel switch (e.g., based upon addressing, protocol,and other information associated therewith) and may implement a flowassignment rule configured to assign the flows 681, 682, 687, and 688 tothe same cluster device (e.g., device 2 614). Accordingly, all of theflows comprising the tunnel switch (681, 682, 687, and 688) may beassigned to device 2 614. The flow assignment rule may prevent the flowsfrom being spread to other devices (e.g., prevent flows 681 and 682 frombeing handled by a first device (device 2 614), while flows 687 and 688are handled by a second device (device 3 616)).

Additional flow assignment rules may be imposed when dealing with IPsecurity tunnels (IP Sec). A single cluster device may be configured toestablish an IPSec tunnel using an IPSec module, which may beimplemented as a kernel module of the device (e.g., in a kernel moduleof the traffic processing module 462 of FIG. 4). The IPSec module may becommunicatively coupled to an IKE module, which may be used to performkey exchange operations used to setup an IPSec session and/or tunnel. Insome embodiments, the IKE module may be implemented as a user spaceprocess (e.g., a user space process of the traffic processing module462). For example, the IKE module may be configured to create Phase IISecurity Associations (P2SA) for use by the IPSec kernel module. TheIPSec module, while handling traffic on the IPSec tunnel managedthereby, may periodically request key updates from the IKE module (for arekey operation), for dead peer detection, as well as termination of aIPSec session or tunnel.

FIG. 7 is a block diagram of one example of a cluster 710 comprising asingle IKE. The cluster 710 includes a single IKE 749 to which each ofthe IPSec modules 747A, 747B, 747C, and 747D of the cluster devices 712,714, 716, and 718 are linked (e.g., via respective cluster interfaceports (not shown) communicatively coupled to a network device, such as arouter, switch, concentrator, or the like). Only the IKE 749 of thecluster master (device 712) may be active. The other devices 714, 716,and/or 718 may include an IKE module (not shown), which may be activatedin the event the device is elected to operate as the cluster master.Accordingly, only the IKE 749 of the cluster master 712 may perform IKEkey exchanges with peer computing devices (not shown). The IKE 749 mayperform all IKE operations for the other cluster devices 714, 716, and718 including, but not limited to: P1SA negotiations, P2SA negotiations,rekey operations, dead peer detection, and the like.

The IKE 749 may generate P2SAs as needed by the cluster devices 714,716, and/or 718 (e.g., in order to establish an IPSec tunnel between thedeice 714, 716, and/or 718 and an external peer). After generating aP2SA for a cluster device (e.g., device 714), the cluster master 712 maytransmit the PS2A thereto. The cluster device may then handle the IPSecflow accordingly (e.g., using the P2SA generated by the IKE 749). Insome embodiments, the cluster master 712 may implement a flow assignmentrule, in which all flows that use a particular P2SA are assigned to aparticular cluster device. For example, if the IKE 749 generates aparticular P2SA for the cluster device 714, and the P2SA is subsequentlyused in another flow, the cluster master may be configured to assign thenew flow to the cluster device 714 (the device that holds the particularP2SA). Similarly, when a flow is reassigned from a first cluster device(e.g., device 714) to a second cluster device (e.g., device 716), thePS2A and other IPSec information relevant to the reassigned flows may betransferred from the cluster master device 712 to the second device(device 716). Following the reassignment, subsequent flows that use thetransferred P2SA may be assigned to the second device.

As shown in FIG. 7, each of the IPSec modules 748B, 748C, and 748D maybe communicatively coupled to the IKE 749. As such, the IPSec modules748B, 748C, and 748D may perform callbacks to the IKE 749 in order tomaintain P2SA sequence number information, perform rekey operations,perform DPD (e.g., DPD hello operations), and the like. Thecommunication link between the IPSec modules 748B, 748C, and 748D may beimplemented using respective cluster interface ports of the devices 714,716, and 718, which may provide for fast and efficient communications.

Since the IKE 749 maintains IPSec data for all of the devices in thecluster 710, the cluster master 712 may be capable of performing moregranular load balancing (all of the devices in the cluster 710 use thesame IKE 749 and, as such, the devices 712, 714, 716, and 718 may“appear” to external peers as a single device for the purposes ofIPSec). For example, IPSec tunnels (such as VPN tunnels or the like) maybe assigned to different devices within the cluster 710. Therefore,although flows that use the same P1SA and/or P2SA information may berestricted to be handled by the same cluster device, other flows usingdifferent key security association information may be spread across thecluster 710 (e.g., the cluster device 714 may handle a first set of VPNconnections, while the cluster device 716 handles a second set of VPNconnections, and so on), including flows associated with the same peer.Moreover, IPSec security protocols, such as DPD, rekey, and the like maybe implemented across all of the devices in the cluster 710.

FIG. 8 is a block diagram depicting a distributed IKE approach. In theFIG. 8 example, each of the devices 812, 814, 816, and 818 in thecluster 810 may implement its own IKE module 851A, 851B, 851C, and 851D.Each IPSec modules 847A-847D communicates with its respective IKE851A-851D. Accordingly, IKE information need not be transmitted betweencluster devices. As shown in FIG. 8, the IPSec modules 847A-847D are notcommunicatively coupled to one another; however, the devices themselves(812, 814, 816, and 818) may be communicatively coupled for otherreasons (e.g., the synchronize cluster configuration data, flow,run-time synchronization, global run-time synchronization data, and thelike).

Since each cluster device 812, 814, 816, and 818 implements its own IKE851A-851 D, the devices may not appear to external peers as a “singledevice” as in the FIG. 7 example. Therefore, the cluster master 812 mayimplement a flow assignment rule specifying that all IPSec flows of aparticular peer be assigned to the same cluster device 812, 814, 816,and/or 818.

FIG. 9A is a block diagram 900 depicting an example of flow assignmentin a cluster comprising a shared IKE module (e.g., the cluster 710 ofFIG. 7). In the FIG. 9A example, a remote peer 933 establishes IPSecflows 981, 982, 983, and 984 with the cluster 910. Since the IPSecmodules 947A-947D of the cluster devices 912, 914, 916, and 918 use acommon IKE 749 (of the cluster master device 912), the cluster 910 mayappear to be a single device to the remote peer 933 for the purposes ofIPSec (e.g., sequence number, rekey, DPD, etc.). Therefore, the clustermaster device 912 may assign flows of the peer 933 to different clusterdevices. The flows 981 and 982 (which may share a common P2SA) may behandled by the cluster device 914, and the flows 983 and 984 (which mayshare a different, common P2SA) may be handled by the cluster device916.

FIG. 9B is a block diagram 901 depicting an example of flow assignmentin a cluster comprising distributed IKE modules (e.g., the cluster 810of FIG. 8). As in FIG. 9A, a remote peer 933 establishes IPSec flows981, 982, 983, and 984 with the cluster 911. The cluster 911 isconfigured to implement distributed IKEs and, as such, each of thecluster devices 912, 914, 916, and 918 implements its own IKE 951A-951D.Accordingly, the cluster devices may not appear as a single device tothe peer 933 for the purposes of IPSec. Accordingly, the cluster master912 may implement a flow assignment restriction rule specifying that allIPSec flows from the peer be handled by the same cluster device. Asshown in FIG. 9B, all of the IPSec flows 981, 982, 983, and 984 arehandled by the cluster device 914. According to the flow assignmentrestriction, the IPSec flows of the peer 933 may not be spread acrossthe cluster devices (e.g., if the device 914 is handling flows 981 and982, the device 916 may not handle flows 983 and/or 984).

A cluster according to the teachings of this disclosure may beconfigured to provide multiple wide area network VPN configurations.Each WAN may have a respective local and remote gateway pairings thatmay be failed over to one another. The local/remote gateway pairs may bedefined in an IKE policy (which may be part of a cluster configuration,security policy, or the like). A flow assignment rule may be implementedto assign all flows associated with a particular IKE policy to the samecluster device (e.g., each cluster device may be responsible for adifferent, respective IKE group policy). Alternatively, the clustermaster (or other device) may implement a plurality of shared IKEmodules, one IKE per IKE group policy. Referring to FIG. 7, the clustermaster device 712 may implement multiple IKE modules 749; one for eachIKE group policy. The IPSec modules 747A-747D of the cluster devices712, 714, 716, and 718 may be configured to synchronize IKE data witheach of the respective IKE modules, on a per-flow basis (e.g., mayselect the IKE associated with the IKE group policy of a particularflow).

FIG. 10 is a flow diagram of another embodiment of a method forassigning flows to cluster devices. The method 1000 may be implementedon a computing device, such as the device 410 of FIG. 4. The method 1000may be implemented using one or more computer-readable and/orcomputer-executable instructions. The instructions comprising the method1000 may be implemented as one or more distinct software modules, whichmay be stored on a computer-readable storage medium, such as a harddisc, optical storage media, memory, or the like. In some embodiments,one or more steps of the method 1000 may be tied to particular machinecomponents, such as computer-readable storage media, communicationsinterfaces, processing modules, or the like.

At step 1010, the method 1000 may start and/or be initialized.Initializing the method 1000 may comprise loading one or morecomputer-readable instructions from one or more computer-readablestorage media, accessing and/or initializing one or more communicationsinterfaces, and the like.

At step 1020, network traffic may be received. The network traffic mayhave been received via a network interface (e.g., interface 120 and/or122 of FIG. 1), such as a hub, switch, router, concentrator, or thelike. If the cluster of method 1000 is operating in “flooding” mode, theinbound traffic may be received by all of the devices in the cluster. Ifthe cluster is operating in “direct forwarding” mode, the traffic may bereceived only by the cluster master device (or the device currentlyassigned to handle the flow).

At step 1030, the cluster master may determine whether the networktraffic is associated with a known flow (e.g., a flow that is beinghandled by one of the devices in the cluster). Step 1030 may compriselooking up the flow in a flow assignment data structure, such as atable, index, or the like. The flow assignment data structure mayprovide a mapping between network flows and the cluster devices assignedthereto. In the data structure, a flow may be identified based upon asource address thereof (IP address, MAC address, or the like), flowprotocol, flow port, flow security information, or the like. The clusterdevice assigned to the flow may be identified according to acluster-specific identifier, MAC address, cluster address (e.g., addressof a cluster interface port of the device), IP address, or the like. Insome embodiments, the flow assignment data structure may furthercomprise flow, run-time synchronization data pertaining to the flow,which may include, but is not limited to: flow security association data(P1SA, P2SA), shared key data, user-session data, flow cache, and thelike. Alternatively, or in addition, the flow assignment data structuremay include a reference (link, pointer, or the like) to the flow,run-time synchronization data associated therewith.

If at step 1030, the method 1000 determines that there is no deviceassigned to handle the flow (e.g., there is no entry for the flow in theflow assignment data structure, the device identifier associated withthe flow entry has not been set, the device that was handling the flowhas been failed over, or the like), the method 1000 may continue at step1040; otherwise, if a device is already actively handling the flow, themethod 1000 may continue at step 1035.

At step 1035, since the network traffic is associated with a flow thathas already been assigned to cluster device that is actively handlingthe flow (e.g., has not failed), the traffic may be ignored by themethod 1000. Accordingly, the method 1000 may terminate at step 1070and/or may continue at step 1020 when additional network traffic isreceived.

At step 1043, one or more device(s) that are available to handle theflow may be identified. The devices may be identified according to a setof one or more flow assignment rules. The flow assignment rules appliedat step 1043 may include, but are not limited to: related flowassignment rules, and security flow assignment rules.

Related flow assignment rules may specify that flows that are related toone another be assigned to the same cluster device. The assignment ofrelated flows to the same device may provide for more efficient flowprocessing, provide for better firewall management (e.g., pin-hole logicfor inbound flow processing), and so on. For example, a related flowassignment rule may specify that all the flows associated with a tunnelswitch connection (discussed above) be assigned to the same device. Theassignment may prevent tunnel switch traffic from being transmittedbetween cluster devices, which may improve the performance of the tunnelswitch (and the cluster generally). Other examples of related flowassignment rules include, but are not limited to: rules to assignforward and reverse flows to the same cluster device (which may providefor improved TCP reset protection), rules to assign related flows to thesame device (e.g., flows associated with the same FTP, VOIP, or similarconnection), and the like. For example, the data and control flows ofthe same FTP connection may be assigned to the same device, which mayprovide for more efficient and secure network traffic and flowprocessing (e.g., protocol inspection logic that opens the data channelpin-hole may be more efficient and secure when implemented on the devicethat is also processing the FTP connection control channel flows).

Security flow assignment rules may perform flow assignment based uponflow security properties. For example, a security flow assignment rulemay cause flows that use the same secure tunnel (e.g., SSH or the like)to be assigned to the same cluster device. As discussed above, theassignment may allow for PS2A sequence number synchronism between flowsto prevent replay attacks and/or to provide for DPD. In another example,a security flow assignment rule may specify that flows using a commonsecurity association (inbound and/or outbound security association) beassigned to the same device. In embodiments implementing a single IKE,security flow assignment rules may allow for load balancing on aper-tunnel basis (e.g., allow flows from the same peer, but usingdifferent secure tunnels, to be handled by different cluster devices).In other embodiments, in which each device implements its own IKE, asecurity flow assignment rule may specify that all secure flowsassociated with a particular peer be assigned to the same clusterdevice.

After applying the flow assignment rules, the method 1000 may continueto step 1047. At step 1047, if (according to the flow assignment rulesapplied at step 1043) only a single device is available to handle thenew flow, the method 1000 may continue at step 1060; otherwise, themethod 1000 may continue at step 1050.

At step 1050, one of the two or more available cluster devicesidentified at step 1043 may be selected to handle the flow. As discussedabove, the selection may be based upon a selection criteria, such asdevice health score, processing load, random selection, round robin, orthe like. After selection of the device to handle the new flow, themethod 1000 may continue at step 1060.

At step 1060, the new flow may be assigned to the identified clusterdevice. Assigning the flow may include, but is not limited to:configuring the identified cluster device to handle network trafficassociated with the flow (e.g., via a configuration message transmittedthereto via a cluster communications port or the like), updating a flowassignment data structure to reflect the flow assignment, configuring anetwork element to direct-forward network traffic related to the flow tothe identified device, and the like.

At step 1070, the method 1000 may end until additional inbound networktraffic is received.

The above description provides numerous specific details for a thoroughunderstanding of the embodiments described herein. However, those ofskill in the art will recognize that one or more of the specific detailsmay be omitted, or other methods, components, or materials may be used.In some cases, operations are not shown or described in detail.

Furthermore, the described features, operations, or characteristics maybe combined in any suitable manner in one or more embodiments. It willalso be readily understood that the order of the steps or actions of themethods described in connection with the embodiments disclosed may bechanged as would be apparent to those skilled in the art. Thus, anyorder in the drawings or Detailed Description is for illustrativepurposes only and is not meant to imply a required order, unlessspecified to require an order.

Embodiments may include various steps, which may be embodied inmachine-executable instructions to be executed by a general-purpose orspecial-purpose computer (or other electronic device). Alternatively,the steps may be performed by hardware components that include specificlogic for performing the steps, or by a combination of hardware,software, and/or firmware.

Embodiments may also be provided as a computer program product includinga computer-readable medium having stored instructions thereon that maybe used to program a computer (or other electronic device) to performprocesses described herein. The computer-readable medium may include,but is not limited to: hard drives, floppy diskettes, optical disks,CD-ROMs, DVD-ROMs, ROMs, RAMs, EPROMs, EEPROMs, magnetic or opticalcards, solid-state memory devices, or other types ofmedia/machine-readable medium suitable for storing electronicinstructions.

As used herein, a software module or component may include any type ofcomputer instruction or computer executable code located within a memorydevice and/or computer-readable storage medium. A software module may,for instance, comprise one or more physical or logical blocks ofcomputer instructions, which may be organized as a routine, program,object, component, data structure, etc., that perform one or more tasksor implements particular abstract data types.

In certain embodiments, a particular software module may comprisedisparate instructions stored in different locations of a memory device,which together implement the described functionality of the module.Indeed, a module may comprise a single instruction or many instructions,and may be distributed over several different code segments, amongdifferent programs, and across several memory devices. Some embodimentsmay be practiced in a distributed computing environment where tasks areperformed by a remote processing device linked through a communicationsnetwork. In a distributed computing environment, software modules may belocated in local and/or remote memory storage devices. In addition, databeing tied or rendered together in a database record may be resident inthe same memory device, or across several memory devices, and may belinked together in fields of a record in a database across a network.

It will be understood by those having skill in the art that many changesmay be made to the details of the above-described embodiments withoutdeparting from the underlying principles of the disclosure.

1. A computer-readable storage medium comprising computer-readableinstructions configured to cause a computing device to perform a methodfor monitoring a cluster comprising a plurality of communicativelycoupled computing devices, the method comprising: within each of thecluster computing devices: calculating a health score of a computingdevice, the health score comprising an indication of the operationalstatus of the computing device, transmitting a status message receivableby the other computing devices in the cluster, the status messagecomprising the health score, and receiving status messages from othercomputing devices in the cluster, each status message comprising acomputing device health score; and a subset of the plurality ofcomputing devices identifying one of the plurality of computing devicesto remove from the cluster in a failover operation based upon the statusmessages.
 2. The computer-readable storage medium of claim 1, whereinthe cluster comprises a computing device configured to operate as acluster master and a computing device configured to operate as a backupmaster, wherein the cluster master computing device is configured tomaintain a global, run-time synchronization data structure comprisingrun-time synchronization data associated with processing tasks assignedto the plurality of computing devices in the cluster, and wherein thecluster master is configured to synchronize the global, run-timesynchronization data structure to the backup master computing device,the method further comprising: transitioning the backup master tooperate as the cluster master when the cluster master is identified tobe removed from the cluster; and selecting another one of the pluralityof computing devices in the cluster to act as the backup master.
 3. Thecomputer-readable storage medium of claim 1, the method furthercomprising, within a subset of the computing devices in the cluster:maintaining run-time synchronization data associated with each of aplurality of tasks implemented by a computing device, and transmittingthe run-time synchronization data to a cluster master computing device.4. The computer-readable storage medium of claim 1, the method furthercomprising: selecting a replacement computing device from the pluralityof computing devices in the cluster to perform a processing taskassigned to the identified computing device; and transitioning theprocessing task to the replacement computing device.
 5. Thecomputer-readable storage medium of claim 4, wherein the selection ofthe replacement computing device is made using the status messages. 6.The computer-readable storage medium of claim 4, wherein the processingtask is associated with run-time synchronization data, the methodfurther comprising transferring the associated run-time synchronizationdata to the replacement computing device.
 7. The computer-readablestorage medium of claim 4, the method further comprising: maintaining atask assignment data structure comprising mappings between clusterprocessing tasks and respective cluster computing devices assignedthereto; and updating the task assignment data structure to map theprocessing task to the replacement computing device.
 8. Thecomputer-readable storage medium of claim 7, the method furthercomprising maintaining a global, run-time synchronization data structurecomprising run-time synchronization data associated with processingtasks implemented by the plurality of computing devices in the cluster.9. The computer-readable storage medium of claim 8, wherein the clustercomprises a computing device configured to operate as a backup master,the method further comprising synchronizing the global, run-timesynchronization data structure to the backup master computing device.10. The computer-readable storage medium of claim 4, wherein thereplacement computing device is configured to operate in standby mode inwhich the replacement computing device is not available to performcluster processing tasks, the method further comprising configuring thereplacement computing device to operate in active mode in which theselected computing device is available to perform cluster processingtasks.
 11. The computer-readable storage medium of claim 10, the methodfurther comprising licensing the replacement computing device to operatein active mode.
 12. The computer-readable storage medium of claim 1,wherein the identification of the computing device to remove from thecluster is based upon a health score thereof.
 13. The computer-readablestorage medium of claim 12, wherein a computing device is identified tobe removed from the cluster when a health score thereof is less than athreshold.
 14. The computer-readable storage medium of claim 12, whereina computing device is identified to be removed from the cluster when ahealth score thereof is maintained less than a threshold for a thresholdtime period.
 15. The computer-readable storage medium of claim 12,wherein a computing device is identified to be removed from the clusterwhen a status message from the computing device is not received within athreshold time period.
 16. The computer-readable storage medium of claim1, wherein the health score comprises an indicator of application-layerperformance of a corresponding computing device.
 17. Thecomputer-readable storage medium of claim 1, wherein the health scorecomprises an indicator of resource usage of the corresponding computingdevice.
 18. The computer-readable storage medium of claim 1, wherein thehealth score comprises indications of application-layer performance,communications interface performance, and device resource status. 19.The computer-readable storage medium of claim 1, the method furthercomprising: generating a health score indicating that the computingdevice should be removed from the cluster responsive to detecting afault in an application module of a computing device; and transmitting astatus message comprising the generated health score.
 20. Thecomputer-readable storage medium of claim 1, the method furthercomprising: generating a health score indicating that the computingdevice should be removed from the cluster responsive to receiving afailover command at a computing device; and transmitting a statusmessage comprising the generated health score.
 21. A system comprising:a cluster communications interface; and a cluster comprising a pluralityof communications devices communicatively coupled to the communicationsinterface, each of the plurality of computing devices comprising acluster monitoring and failover module configured to calculate a healthscore of a computing device, the health score based uponapplication-layer performance characteristics of the computing device,to transmit a status message on the cluster communications interface, toreceive status messages from other cluster computing devices on thecluster communications interface, and to identify a computing device tobe removed from the cluster based using the status messages.
 22. Thesystem of claim 21, wherein one of the plurality of the clustercomputing devices is configured to operate as a cluster master and isconfigured to maintain a global, run-time synchronization data structurecomprising run-time synchronization data associated with processingtasks assigned to the cluster computing devices.
 23. The system of claim22, wherein a subset of the cluster computing devices are configured totransmit run-time synchronization data associated with processing tasksassociated thereto to the cluster master computing device.
 24. Thesystem of claim 23, wherein the cluster master computing device isconfigured to receive run-time synchronization data from the clustercomputing devices and to aggregate the run-time synchronization data inthe global, run-time synchronization data structure.
 25. The system ofclaim 22, wherein the cluster master is configured to select areplacement computing device to perform a processing task assigned tothe identified computing device, the processing task being associatedwith run-time synchronization data, and wherein the cluster mastercomputing device is configured to access the associated run-timesynchronization data in the global, run-time synchronization datastructure and to provide the associated run-time synchronization data tothe replacement computing device.
 26. The system of claim 25, whereinthe cluster master is configured to maintain a processing taskassignment data structure comprising mappings between processing tasksand the cluster computing devices assigned thereto, and wherein thecluster master is configured to update the processing task assignmentdata structure to indicate that the processing task is assigned to thereplacement device.
 27. The system of claim 22, wherein one of theplurality of cluster computing devices, other than the cluster mastercomputing device, is configured to operate as a backup master, andwherein the cluster master computing device is configured to synchronizethe global, run-time synchronization data structure to the backup mastercomputing device.
 28. A method for monitoring a cluster comprising aplurality of communicatively coupled computing devices, the methodcomprising: within each of the cluster computing devices: calculating ahealth score of a cluster computing device, the health score comprisinga status indicator of one or more applications running on the computingdevice, transmitting a status message to a subset of the plurality ofcluster computing devices, the transmitted status message comprising thehealth score, and receiving status messages from other cluster computingdevices, each status message comprising a respective health score, and asubset of the plurality of computing devices identifying one of thecluster computing devices to be removed from the cluster in a failoveroperation based upon the status messages, the failover operationcomprising: selecting a replacement cluster computing device to performa processing task assigned to the identified computing device,transitioning the processing task to the replacement computing device byproviding the replacement cluster computing device with run-timesynchronization data associated with the processing task, and updating aprocessing task assignment data structure to indicate that theprocessing task is assigned to the replacement computing device.
 29. Themethod of claim 28, wherein one of the cluster computing devices is acluster master, and another one of the cluster computing devices is abackup master, the method further comprising: the cluster masteraggregating run-time synchronization data associated with processingtasks assigned to the cluster computing devices in a global, run-timesynchronization data structure; the cluster master synchronizing theglobal, run-time synchronization data structure to the backup master;and the backup master replacing the cluster master when the clustermaster is removed from the cluster.
 30. A method for managing a clustercomprising a plurality of communicatively coupled computing devices by acluster master, the method comprising: receiving a command to modify aconfiguration of each of the cluster computing devices, wherein themodification of a cluster computing device comprises temporarilyremoving the computing device from the cluster; for each of the clustercomputing devices: selecting another cluster computing device toimplement one or more processing tasks of the cluster computing device,transitioning the one or more processing tasks to the selected clustercomputing device, modifying the cluster computing device, and preventingmodification of the selected cluster computing device duringmodification of the cluster computing device.
 31. The method of claim30, wherein the cluster comprises a backup master, the method furthercomprising: selecting another one of the cluster computing devices toreplace the backup master; replacing the backup master with the selectedcluster computing device; modifying the backup master; and preventingmodification of the selected computing device during the modification ofthe backup master.